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1  Executive  Summary 

A  fundamental  shift  in  weather  forecasting  -  from  an  approach  that  is  predominantly  deterministic  to  one 
that  embraces  communication  of  forecast  uncertainty  (i.e.,  probability-based)  -  is  crucial  to  advancing  the 
science  and  application  of  weather  prediction  (NRC  2006).  Probabilistic  forecasts  provide  a  critical  added 
dimension  of  information  that  quantifies  risk  (i.e.,  the  probability  of  undesirable  negative  outcomes)  and 
enables  optimal  decision  making  by  users.  While  there  are  many  approaches  to  generating  probabilistic 
forecasts  (e.g.,  based  on  statistical  analysis  of  past  forecast  errors),  the  most  promising  technique  today 
is  the  combination  of  ensemble  forecasting  with  statistical  post-processing.  Ensemble  forecasts 
dynamically  simulate  the  flow-dependent  forecast  uncertainty  by  varying  initial  conditions  and  model 
processes  across  multiple  runs  of  a  numerical  weather  prediction  model. 

Probability  forecasts  with  high  reliability  and  resolution  can  be  used  effectively  in  decision  making 
processes  to  manage  weather-related  risk  (Eckel  et  al.  2008).  The  importance  of  this  capability  to  the 
success  of  U.S.  military  operations  cannot  be  overstated.  In  the  commercial  marketplace  as  well, 
a  wide  array  of  weather-sensitive  businesses  are  critically  dependent  upon  their  ability  to  effectively 
anticipate  and  manage  weather-related  risk. 

Accordingly,  the  value  of  probabilistic  forecasting  ultimately  lies  in  its  ability  to  help  users  -  whether  they 
are  military  planners,  commodity  traders,  or  families  planning  an  outing  -  manage  weather-related  risk. 

By  quantifying  risk,  probabilistic  forecasting  enables  optimization  of  actions  via  informed  risk  analysis,  in 
which  a  user  evaluates  the  projected  risks  relative  to  their  risk  tolerance  (i.e.,  their  ability  to  absorb  the 
impact  of  the  undesirable  outcome).  To  the  extent  that  the  user’s  risk  tolerance  and  the  potential 
outcomes  can  be  characterized  and  quantified,  these  factors  can  be  combined  with  probabilistic  forecasts 
to  yield  a  decision  support  tool  that  substantially  automates  the  risk  analysis  process. 

The  goal  of  the  Phase  I  R&D  effort  described  herein  has  been  to  lay  a  firm  foundation  for  the 
development  of  such  a  tool.  Toward  that  end,  Impact  Computing  has  teamed  with  the  University  of 
Washington’s  Department  of  Atmospheric  Sciences  to  develop  an  innovative  Weather  Impact 
Probability  Forecast  (WIPCast)  solution  that  implements  and  builds  upon  the  seminal  work  of 
Dr.  Tony  Eckel  in  the  domain  of  probabilistic  forecasting  and  its  practical  application.1 

The  Phase  I  effort  has  yielded  promising  results  that  validate  the  soundness  and  utility  of  our  approach. 

In  particular,  our  working  Phase  I  proof-of-concept  prototype  has  demonstrated  our  ability  to  effectively: 

•  Use  high  resolution,  mesoscale  ensemble  model  outputs  to  produce  forecast  probabilities  of 
meteorological  variables  critical  to  aviation  along  a  particular  flight  path2 

•  Use  calculated  forecast  probabilities  to  determine  anticipated  weather  impacts  on  manned  and 
unmanned  aircraft  flight  routes  (for  a  grid  point,  flight  leg,  and  flight  path)  in  support  of  operational 
risk  assessment  and  management 

•  Provide  an  intuitive  display  of  overall  adverse  weather  impact  calculations  and  probabilities  at  a 
grid  point,  along  a  flight  leg,  and  along  an  entire  flight  path 

This  final  report  is  divided  into  the  following  sections: 

1 .  The  WIPCast  Solution  (Section  2).  An  updated  technical  description  of  our  WIPCast 
methodology  and  solution,  based  on  the  results  of  our  Phase  I  effort. 

2.  Phase  I  Review  (Section  3).  Details  of  our  Phase  I  activities,  results  and  conclusions. 

3.  Phase  I  Option  (Section  4).  Our  plan  for  the  Phase  I  Option  period. 

4.  Phase  II  Plan  (Section  5).  Our  plan  for  a  successful  follow-on  Phase  II  initiative. 


1  Dr.  Eckel  has  been  a  close  and  active  advisor  to  our  team  throughout  the  Phase  I  effort. 

2  The  Phase  I  prototype  has  specifically  focused  on  icing  and  turbulence. 
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2  The  WIPCast  Solution 


The  technical  description  provided  herein  describes  our  WIPCast  solution  as  it  has 
evolved  over  the  course  of  our  Phase  I  effort. 


Our  innovative  approach  to  the  calculation  of  Weather  Impact  Probability  (WIP)  is  at  the  core  of  our 
WIPCast  solution.  WIPCast  integrates  the  rich  informational  content  of  probabilistic  forecasts  with 
mission  impact  analysis  so  as  to  effectively  inform  risk  analysis  (as  a  function  of  risk  tolerance)  to  drive  an 
effective  and  intuitive  decision  support  tool  for  the  end  user.  In  particular,  to  the  extent  that  the  user’s  risk 
tolerance  can  be  characterized  and  quantified,  it  can  be  combined  with  the  risk  outputs  from  probabilistic 
forecasts  to  yield  a  decision  support  tool  that  substantially  automates  the  risk  analysis  process. 

2. 1  Key  Innova  tions  and  Fea  tures 

Key  innovations  and  features  of  our  WIPCast  solution  include: 

•  Calculation  of  Weather  Impact  Probability  ( WIP)  to  convey  the  total  risk  of  serious  degradation 
to  the  complete  mission  from  weather  (see  Section  2.3) 

•  Development  and  application  of  Mission  Impact  Functions  (MIFs)  that  describe  the  uncertainty 
in  the  chance  of  mission  failure  from  actual  occurrence  of  adverse  weather  (see  Section  2.4) 

•  Objective  calculation  of  ambiguity  in  the  ensemble  forecast  and  translation  into  a  WIP 
confidence  interval  to  convey  confidence  in  the  decision  input  (see  Section  2.5) 

•  Statistical  amalgamation  of  the  potential  impact  from  multivariate  weather  sensitivities,  based  on 
a  multivariate  space-time  probability  model  for  the  weather  elements  of  interest,  conditioned 
on  the  calibrated  ensemble  forecasts  (see  Section  2.6) 

•  Intuitive  visualization  and  display  of  anticipated  mission  impacts,  provided  in  the  context  of  a 
comprehensive  weather-based  mission  planning  decision  support  tool  (see  Section  3.2.6) 

•  Flexible  Service  Oriented  Architectures  (SOA)  that  maximizes  interoperability  with  existing 
systems  and  frameworks  (see  Section  5.3.1 1) 

•  Data  source  agnostic  approach  that  avoids  reliance  on  the  peculiarities  or  idiosyncrasies  of  any 
particular  mesoscale  ensemble  data  source  (see  Section  5.3.11) 

2. 2  Limit  a  tions  of  Current  Sta  te-of-the-art 

To  date,  the  common  method  for  incorporating  ensemble  forecast  data  into  risk  analysis  has  been  to 
calculate  the  threshold  probability,  or  chance  of  occurrence  for  a  specific  event,  which  is  described  by  a 
weather  element  and  a  critical  operating  threshold.  This  approach  assumes  that  there  will  be  serious 
degradation  to  the  mission  if  the  weather  occurs  at  or  above  that  threshold,  purely  as  a  step  function. 

The  Tri-Service  Integrated  Weather  Effects  Decision  Aid  (T-IWEDA)  contains  an  extensive  database  of 
such  thresholds,  associated  with  specific  events.  The  critical  threshold  is  defined  as  “the  point  where  the 
occurrence  of  a  meteorological  element  causes  a  significant  (moderate  or  severe)  impact  on  a  military 
operation,  system,  subsystem,  or  personnel”  (Shirkey  and  Gouveia,  2002).  An  example  of  such  a 
threshold  is  surface  winds  >  13  kt  in  a  drop  zone,  where  13  kt  is  considered  critical  for  a  static  line  airdrop, 
since  winds  that  fast  or  higher  can  result  in  serious  injury  to  personnel  and  thus  overall  mission  failure. 

Ensemble  data  (when  calibrated)  can  be  used  to  accurately  compute  the  threshold  probability  [denoted 
as  Pr(wind  speed  >13  kt)].  This  can  be  computed  as  simply  the  relative  frequency  of  forecasts  that 
exceed  the  critical  threshold,  or  by  using  more  complex  methods  such  as  the  “rank  method”  described  in 
Eckel  and  Mass  2005. 
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Figure  1.  Depiction  of  the  threshold  probability,  which  is  the  weather  risk  to  a  static  line  airdrop 

mission  for  an  example  ensemble  forecast  PDF. 

Conceptually,  all  such  methods  estimate  the  area  under  a  continuous  PDF3  (fit  to  the  discrete  ensemble 
forecast  data)  to  the  right  of  the  critical  threshold  (Figure  1 .  ).  Probability  of  exceeding  the  critical 
threshold  of  13  kt  is  found  by  integrating  (i.e.,  summing  up)  the  area  under  the  ensemble  PDF  to  the  right 
of  the  threshold.  This  type  of  calculation  must  be  approximated  from  the  discrete  ensemble  data  since 
the  continuous  PDF  is  normally  not  known.  The  threshold  probability  can  then  be  compared  to  the  user’s 
risk  tolerance  to  arrive  at  an  appropriate  decision  input. 

While  the  approach  of  comparing  threshold  probability  to  risk  tolerance  can  provide  valuable  input  to  the 
decision  process,  it  has  the  following  significant  limitations: 

•  Limitation  #1 :  Use  of  fixed,  deterministic  thresholds.  The  operating  thresholds  in  T-IWEDA  are 
designed  to  be  applied  to  the  type  of  deterministic  (i.e.,  single  valued)  forecasts  that  have  traditionally 
been  supplied  to  DoD  operators.  The  threshold  is  compared  to  the  deterministic  forecast  and  a 
stoplight  chart  decision  input  is  presented  to  the  operator.  When  the  forecast  exceeds  the  critical 
threshold,  the  decision  input  is  red,  meaning  “either  a  total  or  severe  degradation,  or  the  operational 
limits  or  safety  criteria  have  been  exceeded”  (Shirkey  and  Gouveia,  2002).  Since  the  forecast  carries 
a  well  understood  average  amount  of  uncertainty  (i.e.,  the  long-term  average  error),  the  T-IWEDA 
thresholds  were  likely  defined  to  roughly  account  for  that  uncertainty.  Continuing  with  the  above 
example,  the  13  kt  threshold  was  chosen  (hypothetically)  with  the  knowledge  that  the  forecast  has  an 
average  error  of  perhaps  4-5  kt.  The  true  threshold  of  concern  where  injuries  really  begin  to  mount 
may  actually  be  more  like  1 7  kt,  so  in  order  to  hedge  on  the  side  of  caution  given  the  likely  magnitude 
of  forecast  error,  the  critical  threshold  was  set  at  13  kt.  While  that  supposition  could  be  argued 
against,  it  is  clear  that  fixed,  deterministic  thresholds  have  a  questionable  meaning  in  the  context  of 
ensemble  forecasting  where,  instead  of  the  average  uncertainty,  the  dynamic  (or  case-by-case) 
forecast  uncertainty  is  now  known  and  conveyed  with  a  forecast  PDF  (Figure  1 .  ). 

A  revised  methodology  is  required  to  make  best  use  of  ensemble  data  in  calculating  the  risk  by 
introducing  the  concept  of  the  impact  itself  being  uncertain  even  when  the  weather  conditions  are 
known  precisely. 

The  WIPCast  Solution.  WIPCast  addresses  the  problematic  application  of  T-IWEDA  thresholds  with 
ensemble  data  by  introducing  the  concept  of  a  Mission  Impact  Function  (MIF),  which  describes  the 
impact  from  weather  conditions  in  probabilistic  terms  as  opposed  to  fixed,  deterministic  thresholds. 
See  Section  2.4. 

•  Limitation  #2:  Use  of  marginal  operating  thresholds.  T-IWEDA  defines  marginal  operating 
thresholds  to  calculate  marginal  risk  (i.e.,  yellow/amber  stoplight)  as  well  as  negligible  risk  (i.e.,  green 


3  PDF  (Probability  Density  Function)  is  a  function  that  describes  the  relative  likelihood  of  a  particular 
continuous  random  variable  occurring  at  a  given  point. 
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stoplight).  In  our  example,  9  kt  is  the  marginal  threshold  so  9  kt  <  wind  <  13  kt  is  the  marginal 
operating  range  and  wind  <  9  kt  corresponds  to  “no  operational  restrictions”  (Shirkey  and  Gouveia, 
2002).  This  threshold  is  inappropriate  to  apply  to  ensemble  data  for  the  same  reasons  as  described 
above,  but  with  an  additional  complication.  A  yellow  decision  input  should  be  interpreted  as  unclear 
risk,  whereas  green  and  red  mean  a  clearly  acceptable  or  unacceptable  risk,  respectively. 

In  applying  ensemble  data  to  calculation  of  risk,  segregation  of  clear  vs.  unclear  decision  inputs 
requires  introducing  another  new  concept  called  ambiguity.  Ambiguity,  which  refers  to  the  uncertainty 
in  the  probability  forecast,  provides  a  much  more  meaningful  mechanism  for  determining  the 
existence  of  marginal  risk. 

The  WIPCast  Solution.  To  account  for  ambiguity,  WIPCast  calculates  the  ambiguity  in  the  acquired 
ensemble  forecasts  following  the  methods  described  by  Eckel  et  al.  (2011)  and  applies  that 
information  to  calculate  confidence  intervals  for  WIP.  See  Section  2.5. 

•  Limitation  #3:  Oversimplified  handling  of  multivariate  events.  A  mission  may  be  impacted  by 
multiple  events  involving  multiple  weather  elements  with  various  operational  sensitivities  and  different 
relative  importance.  Simply  applying  risk  analysis  on  each  event  separately  can  lead  to  suboptimal, 
or  potentially  even  erroneous,  decisions. 

A  novel  approach  is  required  that  effectively  accounts  for  the  correlation  between  disparate  weather 
elements  so  that  their  associated  risks  can  be  intelligently  analyzed  simultaneously. 

The  WIPCast  Solution.  WIPCast  employs  an  innovative  statistical  approach  to  addressing  the 
complex  multivariate  chalienge  as  described  in  Section  2.6. 

•  Limitation  #4:  Graphical  representation  of  risk  as  a  step  function.  Risk  is  not  a  step  function. 
Solely  using  thresholds  to  make  a  binary  (or  more  precisely  trinary)  determination  -  i.e.,  as  to  when 
risk  goes  from  low  to  moderate  to  high  --  is  often  inadequate,  or  potentially  even  misleading,  in  the 
context  of  an  operational  decision  support  tool.  The  user  needs  to  be  able  to  understand  “how  red  is 
red”  in  order  to  make  a  fully  informed  decision  as  to  whether  or  not  to  take  an  educated  risk. 

It  is  crucial  for  decision  support  tools  used  for  risk  analysis  to  depict  the  severity  of  risk,  not  solely  as 
a  step  function,  but  along  more  of  a  continuum.  Users  need  to  be  able  to  understand  how  far  beyond 
a  moderate  or  high  risk  threshold  their  anticipated  risk  lies. 

The  WIPCast  Solution.  WIPCast  provides  multiple  integrated  risk  dispiays  that  jointly  provide  the 
user  with  deeper  insight  into  their  risk,  helping  them  to  more  precisely  quantify  the  extent  of  the 
anticipated  risk.  See  Section  3.2.6. 

2. 3  Wea  ther  I mpa ct  Probability  (WIP) 

The  basic  concept  of  a  probabilistic  impact,  and  how  to  apply  it  in  calculating  WIP,  was  adapted  from  a 
visionary  Air  Force  publication,  AW S/TR--9 1/001 ,  that  was  unfortunately  never  adopted  into  DoD  weather 
support  operations  (AWS  1991).  The  approach  described  therein  was  to  calculate  a  “mission  success 
indicator”  by  integrating  the  product  of  the  forecast  PDF  and  a  function  describing  the  chance  of  no 
impact.  In  our  case,  rather  than  focusing  on  the  chance  of  mission  success,  we  focus  on  risk  (or  chance 
of  failure),  using  what  we  refer  to  as  the  Mission  Impact  Function  (MIF)  which  is  simply  the  inverse  of  their 
no-impact  function.  MIF  describes  the  impact  from  weather  conditions  in  probabilistic  terms  as  opposed 
to  fixed,  deterministic  thresholds.  (Our  approach  to  calculating  MIF  is  further  described  in  Section  2.4.) 

WIP  may  thus  be  calculated  as: 

WIP  =  J  F(x)  MIF(.t)  dx 
x 

where  x  is  the  continuous  variable  (e.g.,  wind  speed)  and  F  is  the  forecast  PDF. 
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Three  sample  forecasts  and  WIP  results  are  shown  in  Figure  2. .  The  trivial  cases  are  when  the  range  of 
possible  wind  speed  (described  by  the  ensemble  forecast  PDF)  is  so  low  that  no  impact  is  possible  and 
WIP  =  0%  (Figure  2.  a),  or  so  high  that  impact  is  certain  and  WIP  =  1 00%  (Figure  2.  c).  A  threshold 
probability  gives  the  same  risk  estimate  for  these  trivial  cases.  The  interesting  case  that  shows  the 
value  of  WIP  is  when  there  is  overlap  between  the  two  distributions  and  WIP  is  able  to  properly 
convey  the  overall  uncertainty  (Figure  2.  b).  Comparing  Figure  2.  b  to  Figure  1.  reveals  the  limitations 
of  using  threshold  probability  to  describe  risk.  Essentially,  risk  in  Figure  1 .  is  calculated  the  same  way  as 
for  WIP,  but  in  the  place  of  MIF  is  a  step  function  (0  forx  <  13,  and  1  for  x  s  13)  to  represent  the  impact. 
The  resulting  threshold  probability  in  this  example  presents  a  risk  =  21.5%  vs.  the  more  robust  risk  from 
WIP  of  30.2%.  Threshold  probability  tends  to  bias  risk  toward  the  extremes  (0%  and  100%)  by 
underestimating  low  risk  values,  when  the  critical  threshold  falls  on  the  right  side  of  the  forecast  PDF,  and 
overestimating  high  risk  values,  when  the  threshold  falls  on  the  left  side.  This  bias  can  notably  degrade 
the  risk  analysis  and  the  user’s  performance. 


Figure  2.  Sample  calculations  of  WIP  with  the  static  airdrop  MIF  (dashed)  for  three  different 
ensemble  forecast  PDFs  (solid  red)  in  (a)  very  light  wind,  (b)  a  moderately  windy,  and  (c)  high 

wind  situations. 
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A  further  complicating  factor,  effectively  addressed  in  our  approach  to  calculating  WIP,  is  that  of 
ambiguity.  In  particular,  while  the  MIF  calculation  described  in  Section  2.4  properly  represents  the  risk  of 
mission  impact,  it  is  actually  an  expected  value  (or  best-estimate)  of  the  risk  since  there  is  also 
uncertainty  about  the  risk  itself.  That  uncertainty  comes  from  the  fact  that  a  well-calibrated  ensemble 
forecast  PDF  still  contains  random  error  (Eckel  et  al.  2010)  which  carries  over  into  WIP.  This  type  of 
uncertainty  is  termed  ambiguity  since  it  is  2nd  order  uncertainty  as  opposed  to  an  ensemble’s  1st  order 
estimate  in  the  forecast  uncertainty  (NRC  2006). 

To  account  for  ambiguity,  WIPCast  calculates  the  ambiguity  in  the  acquired  ensemble  forecasts  following 
the  methods  described  by  Eckel  et  al.  (2011)  and  applies  that  information  to  calculate  confidence 
intervals  for  WIP.  (The  details  of  our  approach  to  calculating  ambiguity  are  further  discussed  in  Section 
2.5.)  Comparing  the  WIP  confidence  interval  to  the  user’s  risk  tolerance  helps  drive  highly  effective  and 
innovative  decision  support  mechanisms  for  visually  quantifying  risk  for  the  user. 

2.4  Mission  Impact  Function  (MIF) 

WIPCast  addresses  the  problematic  application  of  T-IWEDA  thresholds  with  ensemble  data  by 
introducing  the  concept  of  a  Mission  Impact  Function  (MIF),  which  describes  the  impact  from  weather 
conditions  in  probabilistic  terms  as  opposed  to  fixed,  deterministic  thresholds. 


Figure  3.  Example  MIF,  representing  the  chance  of  impact  to  static  line  airdrop  mission  for  a  given 
observed  wind  speed  as  a  (a)  PDF  or  (b)  CDF.  Solid  lines  in  (b)  indicate  marginal  (9  kt)  and 
critical  (13  kt)  thresholds,  tied  to  the  10th  and  50th  percentile  values  respectively,  help  define  MIF. 

Returning  to  the  static  line  airdrop  mission  example  discussed  in  Section  2.2,  there  is  a  chance  of  impact4 
when  the  observed  wind  is  13  kt,  but  a  greater  chance  of  impact  at  15  kt  and  lower  (but  non-negligible) 
chance  at  10  kt.  This  continuum  is  described  by  a  MIF,  which  can  be  viewed  as  a  probability  distribution 
function  (PDF)  (Figure  3.  a)  but  is  actually  applied  in  the  form  of  a  cumulative  distribution  function  (CDF) 
(Figure  3.  b).  In  this  example,  the  probability  of  impact  increases  rapidly  for  an  observed  wind  from  5  kt  to 

4  “Impact”  as  used  herein  refers  to  a  degradation  of  the  mission  serious  enough  to  cause  mission  failure. 
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13  kt  and  continues  to  rise  at  a  slowing  rate  up  to  25  kt.  For  wind  speeds  above  25  kt,  an  impact  is 
guaranteed,  and  for  wind  <  5  kt  there  is  no  chance  of  impact.  (Note:  This  sample  MIF  is  only  an  initial 
attempt  with  an  unknown  degree  of  fidelity,  formulated  as  described  below.) 

In  the  context  of  military  systems,  the  uncertainty  described  by  a  MIF  comes  from  various  random 
influences  on  weapon  system  performance.  For  our  example,  the  uncertainty  may  come  from  factors 
such  as  the  variability  of  the  terrain  and  its  effects  on  landing,  airborne  personnel  readiness  (jumpers’ 
skill,  experience,  amount  of  fatigue,  etc.),  and  accuracy  of  the  insertion.  The  way  to  interpret  a  MIF  of 
75%,  for  example,  is  that  in  3  out  of  4  drops  with  the  same  conditions  (16  kt  winds,  same  landing  zone, 
etc.),  those  random  factors  cause  a  sufficient  number  of  injuries  for  the  mission  to  fail.  However,  in  1  out 
of  those  4  drops  the  mission  will  “get  lucky”  and  suffer  few  enough  injuries  to  proceed.  Note  that  MIF 
should  not  be  interpreted  as  a  percent,  or  fractional  degradation  to  the  mission. 

2. 5  Calcula ting  Ambiguity 

The  primary  causes  of  ambiguity  are  (a)  limited  sampling  by  an  ensemble,  due  to  the  discrete  number  of 
members,  and  (b)  deficiencies  in  the  ensemble’s  ability  to  simulate  all  sources  of  forecast  uncertainty 
(Eckel  et  al.  201 1 ).  The  effect  of  limited  sampling  alone  can  be  very  significant  for  a  small  ensemble,  as 
displayed  in  Figure  4.  (Allen  2009). 


Figure  4.  (Taken  from  figure  1 ,  Allen  2009)  Demonstration  of  ambiguity  due  to  limited  ensemble 
sampling.  An  8-member  perfect  ensemble  was  simulated  by  sampling  from  a  hypothetical  true 
forecast  PDF  (solid  line),  defined  as  N(0,1).  Ensemble  members  are  represented  with  a 
histogram;  realized,  ensemble  PDF  is  the  dashed  line,  (a)  Ensemble  correctly  represents  the  true 
PDF.  (b)  Ensemble  PDF  has  a  random  error  centered  too  low  with  too  small  a  spread,  (c) 
Ensemble  PDF  has  a  random  error  with  too  high  a  spread. 
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A  perfect  ensemble  that  samples  from  the  true  PDF  will  be  statistically  consistent  over  many  cases  (i.e., 
perfectly  reliable),  but  can  err  notably  in  representing  the  true  PDF  on  any  one  case  and  thus  err  in  the 
estimate  of  forecast  probability  (or  risk)  for  a  specific  event.  Given  an  imperfect  ensemble,  as  in  today’s 
operational  ensemble  systems,  the  problem  is  amplified  since  sensitivities  to  sources  of  forecast 
uncertainty  poorly  simulated  in  the  ensemble  can  vary  case-by-case,  thus  compounding  the  random  error 
in  the  ensemble  forecast  PDF  (Eckel  et  al.  2011). 

Eckel  et  al.  (201 1 )  developed  statistical  methods  to  estimate  ambiguity  and  applied  them  to  operational 
ensemble  forecasts.  Figure  5.  shows  that  ambiguity  can  be  quite  significant  even  for  a  large,  51 -member 
ensemble.  It  also  shows  that  a  sharper  forecast  PDF  (i.e.,  smaller  spread)  is  more  vulnerable  to 
ambiguity  effects  since  a  small  shift  of  the  ensemble’s  PDF  relative  to  the  true  PDF  can  cause  a  large 
error  in  the  ensemble’s  forecast  probability  estimate  (Eckel  et  al.  2011). 


0  10  20  30  40  50  60  70  80  90  100 

Calibrated  Ensemble  Fa  recast  Probability  (%) 


Figure  5.  (Taken  from  figure  13,  Eckel  et  al.  2011)  Ambiguity  represented  as  a  90%  confidence 
interval  about  calibrated,  5-day,  2-m  temperature  forecasts  from  the  51 -member  Japanese 
Meteorological  Agency  operational  ensemble  forecast.  The  dashed  line  is  the  expected  value 
from  the  calibrated  forecasts,  which  demonstrate  perfect  reliability.  The  solid  lines  show  the 
upper  and  lower  bound  of  the  range  of  possible  true  value  of  probability  given  an  ensemble 
spread  of  2.0°C  (thin  line)  and  6.0°C  (thick  line). 

Szczes  (2008)  showed  how  ambiguity  information  is  critical  to  properly  defining  when  a  risk  is  clearly 
acceptable  (green  stoplight)  or  unacceptable  (red  stoplight)  vs.  unclear  (yellow  stoplight).  Basically,  when 
the  confidence  interval  (i.e.,  range  of  possible  values)  about  the  expected  value  of  risk  overlaps  the  user’s 
risk  tolerance,  the  decision  to  take  action  is  unclear  since  the  true  risk  could  go  either  way.  Allen  and 
Eckel  (2011)  further  demonstrated  that  ambiguity  information  can  be  applied  in  the  decision  process  to 
benefit  the  user.  This  involves  consideration  of  secondary  factors  important  to  the  user  that  are  not  easily 
incorporated  into  the  risk  tolerance,  such  as  personnel  morale.  The  idea  is  that  when  the  decision  to  act 
is  unclear,  the  user  may  base  the  decision  on  those  secondary  factors,  thus  maintaining  the  overall 
optimization  of  the  primary  concerns  while  also  optimizing  with  respect  to  the  secondary  ones. 

From  a  decision  support  standpoint,  a  key  outcome  of  our  Phase  I  effort  is  the  redefining  of  marginal  risk 
as  impact  values  within  the  range  of  ambiguity  in  the  risk  (i.e.,  the  risk  “spread”),  as  shown  in  Figure  6.  . 
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Application  of  Ambiguity 


Risk  Acceptable 


Risk  Tolerance  =  35% 


Yellow  =  Appropriate  decision  unclear 
due  to  ambiguity  in  risk,  so 
action  is  optional 


TIME 


Figure  6.  Redefining  marginal  risk  (what  does  “yellow”  mean?) 


2. 6  Multivariate  Risk  Calculation  (Total  WIP) 

So  far,  we  have  limited  our  discussion  to  scenarios  where  WIP  depends  on  a  single  weather  element. 

In  practice,  however,  the  chance  of  an  impact  (and  mission  failure)  depends  on  multiple  elements,  such 
as  wind  speed  and  direction,  visibility,  turbulence,  and  icing.  In  addition,  it  depends  on  these  quantities  at 
multiple  points  in  both  time  and  space,  for  example,  over  an  entire  flight  path.  Thus,  in  practice,  the  total 
WIP  depends  on  a  large  number  of  inter-related  weather  phenomena  simultaneously. 

Although  meeting  the  challenge  of  accurately  calculating  a  total  WIP  that  accounts  for  concurrent  risks 
from  multiple  weather  sensitivities  has  been  deferred  to  Phase  II,  a  proof-of-concept  simplified  approach 
was  employed  in  Phase  I.  In  the  Phase  I  prototype,  WIP  was  generated  separately  for  two  weather 
sensitivities  (turbulence  and  icing)  and  a  total  WIP  was  produced  that  accounts  for  both  risks  using  a 
simplification.  Further  detail  is  provided  in  Section  5.3.3. 


2. 7  Calibra tion  of  Probabilistic  Forecasts 

Today,  the  resolution  of  ensemble  systems  has  increased  sufficiently  for  mesoscale  circulations  to  be 
predicted  probabilistically.  However,  the  output  from  mesoscale  ensemble  systems  should  not  be  used 
directly  in  the  production  of  probabilistic  forecasting  guidance  for  aviation  or  other  uses.  All  forecasting 
systems  possess  systematic  bias  that  does  not  represent  forecast  uncertainty,  much  of  which  can  be 
removed  by  post-processing  (Eckel  and  Mass,  2005).  Post-processing  of  ensemble  member  output  can 
potentially  greatly  improve  the  reliability  and  sharpness  of  the  final  probabilistic  products  derived  from 
ensemble  systems.  Post-processing  can  range  from  simple  bias  correction  (e.g.,  Baars  and  Mass,  2005) 
to  sophisticated  Bayesian  Model  Averaging  (BMA)  approaches  that  include  bias  correction,  weighting  of 
ensembles  based  on  past  performance,  and  adjusting  the  variability  of  each  member  to  produce  optimal 
probability  density  functions  (Raferty  et  al.,  2005). 

WIPCast  needs  an  effective,  efficient  calibration  approach  that  easily  adapts  to  the  wide  range  of  weather 
variables  of  importance  to  aviation.  Many  of  the  more  sophisticated  techniques  do  not  meet  these 
criteria.  WIPCast  Phase  I  therefore  employed  the  simple  shift-and-stretch  calibration  technique  described 
in  Eckel  and  Allen  (2011).  The  1st  moment  bias  is  defined  by  the  mean  error  (ME)  in  the  ensemble  mean: 
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where  M  is  the  total  number  of  individual  verifications  (indexed  by  /'),  is  the  ensemble  mean  and  o,  is 

the  observation.  Calibration  of  the  1st  moment  is  thus  accomplished  by  shifting  all  n  members  by  a  shift 
factor: 


ei  =  ei  +  shift,  factor  for  /  =  !...« 


shift,  factor  =  -  ME - 

The  2nd  moment  bias  is  defined  as  a  fractional  error  following  the  concept  of  statistical  consistency.  For  a 
consistent  ensemble,  the  mean  square  error  of  the  bias-corrected  ensemble  mean  (  MSE ^ )  matches  the 

2  2 

average  ensemble  variance  ( cre  )  over  a  large  verification  dataset,  so  that  cre  / MSE ,  =  1 . 


MSE,  = 


n  +  1 


1  M 

=  —Y 

Mjf 


1  M 

=T I  ^ 


■of 


where  n  is  the  number  of  members  and  ey  is  the  /h  ensemble  member  (Eckel  and  Mass  2005).  The 
fractional  error  (FE)  in  average  ensemble  spread  is  given  by: 


FE  = 


MSE_ 

e 


Calibration  is  thus  accomplished  by  stretching  apart  (or  compressing)  the  1  moment  bias-corrected 
members  about  their  mean  ( e  )  by  a  stretch  factor  to  correct  for  spread  that  is  too  small  (or  large): 


e;  =  e  + 


stretch,  factor  for  /  =  1 . .  .n 


stretch.factor  =  — 
FE 


The  fully  adjusted  members  (e*)  are  then  used  to  produce  calibrated  forecast  probability  following  the 
uniform  ranks  method.  Basically,  forecast  probability  is  found  by  (see  Hamill  and  Colucci  1997  for 
complete  explanation): 


1 .  Identifying  the  rank-ordered  position  of  the  event  threshold  (e.g.  wind  speed  >13  kt)  within  the 
n  +  1  possible  locations  among  the  n  ensemble  members. 

2.  Summing  probability  from  all  ranks  above  the  threshold,  where  each  rank  contains  probability  of 
1/(n  +  1). 

3.  Adding  in  a  linear  fraction  of  the  probability  from  the  rank  in  which  the  event  threshold  occurs. 

Besides  producing  continuous  values  of  probability,  this  technique  also  alleviates  biasing  toward  extreme 
probability  that  occurs  with  the  relative  frequency  technique  (Eckel,  2003). 


2. 8  Wea ther  Observa tion  Da  ta  (“Ground  Truth”) 

For  training  the  calibration  and  ambiguity  routines,  and  for  verification  studies,  WIPCast  needs  high 
quality  weather  observations  of  all  forecast  variables  of  interest  to  aviation.  This  includes  not  only  surface 
information  such  as  winds,  visibility,  cumulative  precipitation,  etc.  for  airfield  and  surface  operations,  but 
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also  variables  throughout  the  volume  of  the  atmosphere  such  as  winds,  temperature,  and  relative 
humidity.  Furthermore,  the  data  needs  to  continuously  cover  large  operations’  domains.  A  gridded 
analysis  of  better  or  comparable  resolution  to  the  forecast  data  is  the  best  solution,  but  is  unfortunately 
not  always  available.  The  methodology  employed  for  the  Phase  I  proof-of-concept  is  described  in  Section 
3.2.8  and  our  planned  approach  for  Phase  II  is  discussed  in  Section  5.3.8. 
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3  Phase  I  Review 

As  detailed  in  the  subsections  that  follow,  our  Phase  I  effort  has  proven  highly  productive,  yielding  results 
that  will  serve  as  a  firm  foundation  for  a  follow-on  Phase  II  initiative. 

3. 1  Phase  I  Objectives  and  A ccomplishments 


Our  Phase  I  initiative  has  focused  on  development  of  a  proof-of-concept  prototype,  through  which  ajl 
technical  goals  for  Phase  I  have  been  successfully  achieved  and  demonstrated,  as  listed  in  0. 


Phase  1  Objectives 

Phase  1 

AS  STATED  IN 

SBIR  Solicitation 

AS  STATED  IN 

Phase  1  Proposal 

Accomplishments 

Develop  approaches  and  methodologies  for 
using  mesoscale  ensemble  modeling  to 
produce  forecast  probabilities  of  meteorological 
variables  critical  to  aviation. 

•  Use  high  resolution, 
mesoscale  ensemble 

✓ 

Establish  capability  to  calculate  forecast 
probabilities  for  meteorological  parameters 
using  ensemble  model  forecasts  for 
determination  of  weather  impacts  on  manned 
and  unmanned  aircraft  flight  routes. 

model  outputs  to  produce 
forecast  probabilities  of 
meteorological  variables 
critical  to  aviation  (e.g., 
such  as  icing,  turbulence, 
thunderstorms,  winds,  and 
clouds)  along  a  particular 
flight  path. 

•  Use  calculated  forecast 
probabilities  to  determine 
anticipated  weather 
impacts  on  manned  and 
unmanned  aircraft  flight 
routes  (for  a  grid  point, 
flight  leg,  and  flight  path)  in 

✓ 

Determine  technical  feasibility  and  develop 
approaches  to  producing  probabilistic  forecasts 
of  specific  aviation  weather  variables  such  as 
icing,  turbulence,  thunderstorms,  winds,  and 
clouds  from  mesoscale  ensemble  model  output 
for  aviation  meteorology. 

s 

Establish  a  method  (or  methods)  of  calculating 
ensemble  model  forecast  probabilities  for 
aviation  meteorological  parameters  and  apply 
these  methods  to  individual  parameter 
probabilistic  predictions. 

s 

Develop  an  initial  capability  to  take  one 
parameter’s  adverse  weather  threshold 
probability  of  occurrence  and  translate  that  to 
an  adverse  weather  impact  display  for  a  grid 
point,  flight  leg,  and  an  entire  flight  route. 

support  of  operational  risk 
assessment  and 
management. 

s 

Develop  an  initial  capability  to  take  one 
parameter’s  adverse  weather  threshold 
probability  of  occurrence  and  translate  that  to 
an  adverse  weather  impact  display  for  a  grid 
point,  flight  leg,  and  an  entire  flight  route. 

•  Provide  an  intuitive  display 
of  overall  adverse  weather 
impact  calculations  and 
probabilities  at  a  grid  point, 
along  a  flight  leg,  and  along 
an  entire  flight  path. 

s 

Phase  I  Objectives  vs.  Accomplishments 
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3. 2  Phase  I  Prototype 

Development  of  our  Phase  I  prototype  has  been  a  highly  productive  and  educational  exercise,  yielding 
the  following  key  benefits: 

1 .  Served  as  a  focal  point  for  our  discussions  with  the  customer  and  an  interactive  mechanism  for 
better  communicating  and  demonstrating  some  of  our  key  portrayal  concepts. 

2.  Focused  our  efforts  on  achieving  demonstrable,  measurable  results. 

Our  Phase  1  prototyping  efforts  have  yielded  a  preliminary  implementation  of  the  WIPCast  processing 
pipeline  including: 

•  Calculation  of  Weather  Impact  Probability  (WIP) 

•  Development  and  application  of  a  Mission  Impact  Functions  (MIF) 

•  Objective  calculation  of  ambiguity  in  the  ensemble  forecast  and  translation  into  a  WIP 
confidence  interval  to  convey  confidence  in  the  decision  input 

•  Statistical  amalgamation  of  the  potential  impact  from  multivariate  weather  sensitivities  by 
extending  WIP  calculation  to  use  of  a  Joint  Predictive  PDF  decision  input 

•  WIPCast  displays  that  intuitively  and  clearly  present  anticipated  adverse  weather  impacts  and 
risks  at  a  grid  point,  along  a  flight  leg,  and  along  an  entire  flight  path 

The  proof-of-concept  implementation  focused  on  icing  and  turbulence  as  examples  of  specific  aviation 
weather  variables  of  interest. 

3.2.1  Derived  Weather  Parameters 

Forecasts  for  two  weather  phenomena  were  considered  in  Phase  I  (i.e.,  en  route  turbulence  and  icing) 
and  generated  from  the  raw  WRF  model  output  in  post-processing  algorithms.  A  turbulence  index  and  an 
icing  index,  derived  separately  on  each  ensemble  forecast  member,  were  used  to  represent  the  intensity 
of  each  phenomenon  on  a  continuous  scale.  Fixed  threshold  values  of  the  indices  were  used  to  demark 
the  standard  light  (LGT),  moderate  (MDT),  and  severe  (SVR)  intensity  levels  as  applicable  to  aviation. 

The  turbulence  index  (Tl)  algorithm  followed  the  method  presented  by  Ellrod  and  Knapp  (1992),  which 
considers  the  wind  shear,  deformation  and  convergence  in  the  forecasted,  three-dimensional  model  wind 
field.  Given  the  UWME  model  resolution  of  12km,  we  followed  the  Air  Force  Weather  Agency’s  (AFWA’s) 
empirically  based  thresholds  for  mesoscale  models  of  LGT  (3  <  77  <  9),  MDT  (9  <  77  <  14),  and  SVR  (14  < 
Tl).  This  produced  good  results  at  upper  levels  (>1 0,000ft)  but  often  considerably  over-forecasted 
turbulence  in  the  low  levels.  This  issue  can  be  alleviated  by  implementing  the  Panofsky  Index  (PI)  to 
represent  turbulence  in  the  lower  levels  (Boyle  1990,  Passner  2000,  Brooks  and  Oder  2004).  There  was 
insufficient  time  in  WIPCast  Phase  I  for  this  implementation,  so  the  Phase  I  turbulence  forecasts  have  a 
recognized  high  bias  in  the  lower  levels.  Implementing  the  PI  is  a  task  for  WIPCast  Phase  II. 

The  icing  index  (Icgl)  algorithm  followed  the  method  developed  by  AFWA  as  part  of  the  Joint  Ensemble 
Forecast  System  (JEFS)  experiment  and  demonstration.  The  //value  (Figure  7. )  is  based  on  an 
empirical  formula  that  uses  temperature  (T,  in  Kelvin)  and  relative  humidity  (RH,  in  %)  at  a  specific  model 
grid  point.  The  standard  thresholds  are  LGT  (58  <  Icgl  <  83),  MDT  (83  <  Icgl  <  90),  and  SVR  (90  <  Icgl). 

Icgl  =  TANH(R/-/*6  -  4)  *  TANH((T -  247)/10)*100 
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Figure  7.  Icing  index  (Icgl)  values  with  shaded  levels  of 
LGT  (yellow),  MDT  (orange),  and  SVR  (red)  icing  intensity  levels. 


3.2.2  Ensemble  Forecast  Calibration 

Since  WIPCast  Phase  I  dealt  with  derived  parameters,  there  was  a  choice  between  calibrating  the  state 
variables  used  in  deriving  the  final  parameters  (e.g.,  wind  components  for  77)  or  directly  calibrate  the  final 
parameters.  We  chose  the  latter  to  avoid  the  complications  presented  by  possible  correlations  in  the 
state  variables  involved  in  the  77  and  Icgl  calculation.  Shift  and  stretch  factors  for  a  particular  forecast 
were  computed  using  the  previous  30-days  of  forecasts/observations  as  the  training  period  in  order  to 
identify  seasonally  dependent  systematic  errors.  The  factors  were  broken  out  by  vertical  level  (25 
pressure  levels)  since  biases  can  be  very  different  in  the  boundary  layer  vs.  the  free  atmosphere.  The 
factors  were  also  broken  out  by  forecast  lead  time  since  the  ensemble  PDF’s  systematic  errors  can  be 
highly  dependent  on  the  increasing  error  growth  during  the  forecast  cycle. 

For  Tl,  we  ignored  the  1st  moment  correction  which  was  not  meaningful  to  identify  in  Phase  I  because  of 
the  limitation  in  Tl  calculation  described  above.  Figure  8.  shows  the  stretch  factors  for  Tl  for  two  forecast 
lead  times.  The  results  here  indicate  the  ensemble  is  underspread  (stretch  factors  >  1.0)  as  is  normally 
the  case  for  an  ensemble.  Also  note  the  natural  finding  here  that  a  longer  lead  time  generally  has  a  lower 
stretch  factor  as  the  ensemble  PDF  gets  closer  to  saturating  to  climatology. 
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Figure  8.  Stretch  factors  for  TURBC  and  ICG  for  forecast  hours  32  and  33. 

For  Icgl,  a  significant  bias  was  found  which  could  not  be  ignored.  Furthermore,  the  bias  was  found  to  be 
heavily  dependent  on  the  Icgl  index  itself  so  we  implemented  a  conditional  correction  as  shown  in 
Figure  9.  where  the  shift  factor  is  a  function  of  Icgl.  For  the  ICG  (icing)  stretch  factors,  Figure  9.  shows  a 
much  larger  necessary  stretching  compared  to  TURBC  (turbulence)  likely  due  to  the  ensemble’s  difficulty 
in  simulating  the  range  of  forecast  errors  for  moisture. 


0  5  10  15  20  25  30  35  40  45  50  55  60  65  70  75  80  85  90  95  100 

Icing  Index 


Figure  9.  Shift  factors  for  ICG  at  the  1000  and  725  hPa  data  levels  for  forecast  hour  32. 


There  is  a  notable  complication  in  applying  shift-and-stretch  to  non-Gaussian  variables,  which  we  are 
dealing  with  in  WIPCast.  Tl  is  a  positive  definite  variable  (bound  by  zero  on  the  left)  and  appears  to  follow 
the  gamma  distribution: 


for  x  >  0  and  a,j3  >  0 
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r(a)  = 

Where  x  is  the  random  variable  and  a  and  flare  the  gamma  parameters.  Icgl  values  exist  between  the 
interval  of  0  to  100  so  likely  follow  the  beta  distribution: 

f{x;p,q)  =  y— — t- xp~x (l  —  x)^-1  for0<x<l  and  p,q>0 

B  (p,q) 

B (x\p,q)=[  tp~x (l-t)q]dt 

Jo 

where  p  and  q  are  the  beta  parameters.  The  problem  for  shift-and-stretch  is  that  adjusting  the  two 
moments  of  the  PDF  becomes  intertwined  since  a  shift  can  affect  the  2nd  moment  when  accounting  for 
non-physical  index  values  (negative  Tl  or  0  >  Icgl  >  100)  and  likewise  a  stretch  can  affect  the  1st  moment. 

The  solution  is  to  apply  the  corrections  on  e  and  cr,  as  if  the  PDF  were  Gaussian  but  find  the  calibrated 
member  values  ( e  *  )  using  the  PDF’s  natural  distribution  following  these  steps: 

1)  Fit  the  natural  PDF  to  the  raw  ensemble  members  using  the  method  of  moments 

2)  Find  the  original  percentile  of  each  ensemble  member  using  the  fitted  PDF. 

3)  Fit  the  natural  PDF  to  the  shifted  and  stretched  ensemble  mean  and  standard  deviation  using 
the  method  of  moments.  (Note:  When  the  ensemble  mean  falls  out-of-bounds,  it  must  be 
reset  to  the  extreme  limit.  This  does  impact  the  calibration  but  has  no  impact  on  the  final 
forecast  probability  or  WIP  since  it  represents  a  tiny  adjustment  to  an  extreme  value.) 

4)  Produce  the  calibrated  members  by  finding  the  parameter  values  on  the  shifted-and-stretched 
natural  PDF  associated  with  the  original  percentiles. 

We  illustrate  this  process  in  Figure  10.  with  the  following  example  ICG  forecast  at  a  grid  point  at  the  725 
hPa  level  at  the  32h  lead  time  (i.e.,  these  are  the  nine  raw  Icgl  values): 

EF  =  (92.6,  92.9,  93.2,  83.9,  84.8,  90.2,  94.4,  86.1,  93.4) 

This  EF  is  fit  to  a  beta  PDF  by  plugging  the  mean  (e  =  90.2)  and  spread  (cr,  =  4.12)  into  the  equations 
for  the  method-of-moments  approximation  of  the  beta  parameters: 

(90.y(l-90.2)_902  =  422 
(4.12)2 

q-&± O'2*  =  5.05 
90.2 

Next,  we  find  the  percentile  location  of  each  member  on  the  fitted  beta  PDF 
(0.693,  0.723,  0.752,  0.079,  0.106,  0.453,  0.856,  0.159,  0.770) 

Now  we  find  the  calibrate  beta  PDF  using  the  adjusted  mean  and  spread: 
shifted  mean  =  90.2  +  (-12.33)  =  77.84 

shifted  spread  =  4.12*(1.457)  =  6.00 

which  yields  p  =  36.5,  q  =  10.4  by  the  same  method  above.  Finally,  the  values  for  the  calibrated 
members  are  found  using  the  original  percentiles  on  the  calibrated  PDF: 

EF  calibrated  =  (81.2,  81.7,  82.2,  69.0,  70.1, 77.5,  84.2,  71.8,  82.5) 
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These  values  can  then  be  used  to  produce  calibrated  forecast  probability.  We  will  also  see  later  that  the 
full  calibrated  beta  PDF  will  be  used  in  the  WIP  calculation. 


Figure  10.  Demonstration  of  the  shift-and-stretch  calibration  process  on  a  32-h  ICG  forecast  at  725 
hPa.  The  original  nine  forecasts  of  the  ensemble  data  are  shown  by  the  lower  red  lines  and  the 
final,  calibrated  forecasts  are  the  higher,  blue  lines. 

3.2.3  WIP  Calculation 

WIP  was  calculated  separately  for  TURBC  and  for  ICG  at  every  model  grid  point  (horizontal  and  vertical) 
and  lead  time  using  the  default  MIFs  and  the  calibrated  ensemble  forecast  PDFs.  Figure  1 1 .  shows  a 
sample  result  for  the  risk  to  the  mission  from  ICG  using  the  example  data.  Note  that  the  calibrated  WIP  of 
94.6%  would  have  been  100%  if  the  uncalibrated  forecast  PDF  (Figure  11.)  were  applied  instead. 

For  Phase  I,  we  simply  assumed  that  the  risk  from  TURBC  is  completely  independent  from  the  risk  from 
ICG,  which  provided  reasonable  results  for  most  cases.  The  total  (or  multivariate)  WIP  was  therefore 
found  by: 

WIP total  =  l/WPTURBC  +  WIP\Cq  -  WIPJURBC  *  WIP, CG 
Our  planned  Phase  II  approach  for  more  robust  calculation  of  total  WIP  is  discussed  in  Section  5.3.3. 


Icing  Index /100 

Figure  11.  Sample  calculation  for  risk  from  ICG  that  yielded  a  WIP  =  94.6%. 
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3.2.4  Calculating  Ambiguity  -  Randomly  Calibrated  Resampling 

WIPCast  output  includes  90%  confidence  intervals  about  the  best-estimate  values  of  forecast  probability 
and  WIP  based  on  the  ambiguity  in  the  ensemble  forecast.  For  Phase  I,  we  targeted  implementation  of 
ambiguity  calculation  by  the  randomly  calibrated  resampling  (RCR)  method  since  it  is  more  skillful 
compared  to  the  calibrated  error  sampling  (CES)  method  (Eckel  and  Allen  2011).  RCR  was  successfully 
coded  and  run  but  in  an  incomplete  mode  due  to  the  resource  constraints  of  Phase  I.  Firstly,  the  random 
calibration  component  of  RCR  was  omitted  so  the  ambiguity  is  based  solely  on  resampling  and  is  thus 
likely  an  underestimate.  Secondly,  performing  RCR  on  the  extensive  amount  of  forecast  data  proved  to 
be  extremely  costly  in  computer  processing  time.  To  make  the  WIPCast  runs  manageable,  resampling  at 
each  grid  point  for  each  forecast  variable  was  performed  with  only  100  samples  instead  of  the  desired 
number  of  samples  on  the  order  of  1,000. 


3.2.5  Mission  Impact  Function  (MIF) 

Default  MIFs  for  the  chance  of  impact  from  observed  TURBC  and  ICG  intensities  were  subjectively 
designed  based  on  likely  sensitivities  of  rotary-wing  aircraft  (Figure  12. ). 


ICG  MIF  for  Rotary-Wing  Aircraft 


icing  Index /1 00 


TURB  MIF  for  Rotary-Wing  Aircraft 


Turbulence  Index 


Figure  12.  Default  MIF  for  ICG  and  TURBC.  Standard  thresholds  for  light  (LGT),  moderate  (MDT), 

and  severe  (SVR)  intensities  are  shown  for  reference. 


These  MIFs  are  mainly  intended  to  demonstrate  WIPCast  processing  and  produce  a  reasonable  result. 
The  TURBC  MIF  is  defined  by  a  gamma  CDF  with  //=  9.0  and  <r  =  3.0,  which  equates  to  gamma 
parameters  of  a  =  9.0  and  /?=  1.0  using  the  method  of  moments.  The  ICG  MIF  is  defined  by  a  beta  CDF 
with  /u  =  0.65  and  u  =  0.05,  which  equates  to  beta  parameters  of  p  =  58.5  and  q  =  31.5  also  by  the 
method  of  moments. 


A  key  question  being  addressed  as  part  of  WIPCast  development  is  how  to  appropriately  define  a 
meaningful  MIF  for  each  weather  vulnerability.  A  reasonable  approach  is  to  assume  a  basic  shape  of  the 
PDF  (Gaussian  or  gamma)  and  then  define  its  first  two  moments  (mean  and  standard  deviation)  by 
assigning  assumed  percentile  values  to  the  T-IWEDA  marginal  and  critical  thresholds,  thus  grounding  the 
MIF  on  known  information.  This  process  was  shown  graphically  in  Figure  3.  b  where  the  marginal 
threshold  was  placed  at  the  10th  percentile  and  the  critical  threshold  was  placed  at  the  50th  percentile. 


3.2.6  Decision  Support  /  User  Interface 

Our  Phase  I  effort  has  incorporated  preliminary  prototyping  of  WIPCast  displays  that  intuitively  and  clearly 
present  anticipated  adverse  weather  impacts  and  risks  at  a  grid  point,  along  a  flight  leg,  and  along  an 
entire  flight  path.  The  client  interface  also  provides  a  mechanism  for  the  user  to  characterize  their  risk 
tolerance,  which  then  helps  drive  the  information  presented  in  the  resulting  decision  support  displays. 

The  Phase  I  prototype  user  interface  is  shown  in  Figure  13.  . 
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Figure  13.  WIPCast  Phase  I  Prototype  User  Interface 
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Geospatial  (Aerial)  View.  Provides  color-coded  display  of  operational  risk  along  entire  flight  path. 
Fully  integrated  with  Google  Maps. 

Vertical  Cross  Section  (Altitude  View).  Provides  color-coded  grid  cell  display  of  operational  risk 
at  all  altitudes  along  entire  flight  path,  as  well  as  displaying  terrain  along  entire  flight  path. 

WIP  Graph.  Plots  calculated  WIP  spread  (variance)  along  entire  flight  path.  Graphically  depicts 
relationship  between  WIP  values  and  user-selected  risk  tolerance.  Helps  answer  the  question  of 
“how  red  is  red”. 

Flight  Point  Data  Display.  Provides  detailed  location  (lat/lon/alt)  and  weather  data  for  each 
defined  step  along  the  flight  route. 

User  Controls.  Allow  for  interactive  selection  of  mission  start  time,  flight  route  (path  /  altitude), 
weather  parameters  of  interest,  and  operational  risk  tolerance.  Advanced  controls  are  also 
available  for  adjusting  impact  thresholds  (as  input  to  MIF). 


All  views  are  fully  integrated  and  synchronized,  updating  dynamically  in  response  to  changes  in  user 
selections  (e.g.,  risk  tolerance,  mission  start  time,  etc.). 


3.2.7  Mesoscale  Ensemble  for  Phase  I 

While  the  WIPCast  solution  is  being  carefully  architected  to  avoid  reliance  on  a  particular  ensemble 
system,  the  Phase  1  proof-of-concept  prototype  makes  use  of  the  high-resolution  (12-km)  University  of 
Washington  Mesoscale  Ensemble  (UWME),  a  mesoscale  probabilistic  testbed  that  has  run  in  real-time  for 
nearly  a  decade.  Although  UWME  does  have  known  limitations  (such  as  being  limited  to  nine  members 
and  not  addressing  model  uncertainty),  it  is  still  a  useful  basis  for  the  Phase  1  prototype  given  its  high 
resolution.  Moreover,  UMWE  has  provided  data  for  Phase  I  covering  the  Pacific  Northwest,  a  region  of 
substantial  meteorological  contrasts,  with  a  wide-range  of  surface  conditions  and  climate  types  that 
enabled  us  to  better  explore  the  sensitivities  of  WIP  during  Phase  I.  Our  open  WIPCast  architecture 
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retains  the  option  in  Phase  II  of  ingesting  ensemble  data  from  other  sources,  such  as  the  Air  Force 
Weather  Ensemble  Prediction  System  (AFWEPS)  run  by  the  Air  Force  Weather  Agency,  the  primary 
supplier  of  weather  information  to  the  US  Army. 

UWME  is  an  ensemble  of  nine  short-range  numerical  weather  prediction  (NWP)  model  forecasts  run 
operationally  at  the  UW  Department  of  Atmospheric  Sciences.  Forecasts  are  run  twice  per  day  (at  the  00 
and  12  UTC  cycles)  for  the  0-72  hour  forecast  period  to  produce  skillful  probabilistic  mesoscale 
meteorological  forecasts  for  the  Pacific  Northwest  United  States,  using  the  Weather  Research  and 
Forecasting  (WRF)  model. 

The  forecasts  feature  an  outer  grid  (151x127)  of  36  km  horizontal  grid 
spacing  that  covers  much  of  western  North  America  and  the 
northeastern  Pacific  and  a  nested  grid  (163x124)  of  12  km  grid  spacing 
that  covers  the  entire  Pacific  Northwest  U.S.,  from  central  California  to 
central  British  Columbia,  and  from  600  miles  off  the  west  coast  to  central 
Montana,  as  shown  here.  The  model  utilizes  37  vertical  eta  levels,  and 
is  non-hydrostatic,  which  limits  pressure  gradient  force  errors  in  complex 
terrain.  An  upper-radiative  boundary  condition  is  used  to  allow  gravity 
waves  to  radiate  through  the  model  top  without  being  reflected.  The 
sub-grid  scale  parameterizations  include: 

•  Yonsei  University  planetary  boundary  layer  scheme 

•  Thompson  explicit  moisture  scheme  including  mixed  phase  processes 

•  Kain-Fritsch  cumulus  parameterization 

•  Simple  shortwave  and  RRTM  longwave  radiation  schemes 

•  Noah  land  surface  model 

UWME  employs  a  multi-analysis  strategy  to  provide  the  forcing  of  its  nine  members.  Initial  conditions 
(ICs)  and  lateral  boundary  conditions  (LBCs)  for  the  WRF  mesoscale  ensemble  are  interpolated  from 
separate  synoptic-scale  model  analysis  and  forecast  fields  obtained  from  several  operational  weather 
prediction  centers  worldwide.  In  addition  to  providing  the  LBC  updates,  nudging  by  the  synoptic-scale 
forecasts  is  also  applied  to  the  36  km  (outermost)  model  domain  to  ensure  consistency  with  the 
prescribed  synoptic  forcing.  The  nine  synoptic-scale  ICs  and  LBCs  includes: 

1 .  The  National  Center  for  Environmental  Prediction  (NCEP)  Global  Forecast  System  (GFS) 

2.  The  Canadian  Meteorological  Centre  (CMC)  Global  Environmental  Multiscale  (GEM)  model 

3.  The  National  Center  for  Environmental  Prediction  (NCEP)  Eta  model 

4.  The  Bureau  of  Meteorology  (Australia)  Global  Analysis  and  Prediction  (GASP)  model 

5.  The  Japan  Meteorological  Agency  (JMA)  Global  Spectral  Model  (GSM) 

6.  The  Fleet  Numerical  Meteorology  and  Oceanography  Center  (FNMOC)  Navy  Operational  Global 
Analysis  and  Prediciton  System  (NOGAPS) 

7.  The  Central  Weather  Bureau  (CWB)  (Taiwan)  Global  Forecast  System 

8.  The  United  Kingdom  Met  Office  (UKMO)  Unified  Model  (UM) 

9.  Centroid  (CENT)  -  average  of  the  above  8  analyses/forecasts 

Forecasts  are  computed  on  a  16-node  (116  Intel  Xeon  2.3GHz  processors)  Linux  PC  Beowulf  cluster. 
Each  36/12  km  72-hour  forecast  finishes  in  approximately  2  hours.  Each  global/synoptic-scale  1C 
becomes  available  at  a  different  time  after  0000/1200  UTC.  Thus,  some  UWME  ensemble  members  finish 
before  others.  UWME  processing  is  completed  by  1200/0000  UTC  (0400/1600  PST)  provided  all 
initializations  are  in  on  time. 
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3.2.8  Data  Collection  and  Ingest 

The  University  of  Washington  Mesoscale  Ensemble  (UWME)  was  used  as  the  source  of  both  the 
ensemble  forecast  data  and  the  ground  truth  (i.e.,  verifying  observations)  for  WIPCast  Phase  I.  The  raw 
ensemble  model  output  data  was  vertically  interpolated  from  eta  levels  to  fixed  pressure  levels,  with  25 
mb  vertical  resolution  from  1000  to  700  hPa,  and  50  hPa  resolution  from  700  to  100  hPa.  The  archive  of 
ensemble  data  for  the  WIPCast  Phase  I  includes  surface  fields  and  variables  describing  the  basic  state  of 
the  atmosphere  at  pressure  levels,  including  wind,  temperature,  humidity,  and  geopotential  height.  The 
archive  extends  from  December  21,  2010  through  present. 

While  it  is  best  to  use  a  high  quality  independent  source  for  ground  truth  data  (used  for  calibration  and 
verification),  WIPCast  Phase  I  used  short-term  forecasts  from  UWME’s  Centroid  member  as  ground  truth 
in  order  to  stay  within  budget  constraints.  For  example,  a  UWME  forecast  with  a  lead  time  of  28h  was 
verified  against  the  4-h  Centroid  forecast  from  the  most  recent  forecast  initialization.  This  approach  has 
some  validity  since  updated,  short-term  forecasts  should  be  closer  to  reality  than  forecasts  with  greater 
lead  times,  but  it  of  course  suffers  from  the  fact  that  short-term  forecasts  can  still  carry  significant  errors. 
Also,  the  model’s  systematic  errors  are  shared  between  both  the  forecast  and  ground  truth  so  it  is  difficult 
to  properly  calibrate  the  forecasts.  Note  that  these  limitations  do  not  impact  the  fundamental  soundness 
of  WIPCasts  processing  algorithms  which  are  designed  independent  of  the  quality  of  the  ground  truth.  A 
more  robust  ground  truth  can  easily  be  plugged  into  the  framework. 

3.2.9  Technical  Implementation 

The  WIPCast  prototype  has  been  carefully  designed  to  serve  as  a  firm  foundation  for  development  of  the 
operational  system  in  Phase  II. 

The  front-end  developed  by  Impact  Computing  has  been  written  entirely  in  Java  and  is  therefore  cross¬ 
platform.  Impact  believes  in  employing  open  source  software  where  feasible  and  appropriate  to  avoid 
reinventing  the  wheel.  Accordingly,  two  open  source  projects  -  the  NetCDF-Java  library  and  the 
JFreeChart  library  -  were  leveraged  in  development  of  the  Phase  I  prototype.  (As  part  of  the  Phase  I 
development  effort,  Impact  identified  bugs  in  the  JFreeChart  package  for  which  Impact  submitted  fixes  to 
the  JFreeChart  open  source  project  on  SourceForge.) 

The  back-end  software  developed  by  University  of  Washington  has  been  written  in  Fortran  and  Perl, 
running  on  a  Linux  cluster  with  16  nodes  and  a  40  TB  archive  disk  array. 


3.3  Phase  I  Tasks 

The  specific  tasks  performed  during  Phase  I  included  the  following: 

Task  1.  Ingest  and  processing  of  mesoscale  ensemble  model  output 

•  Captured  high-fidelity  archive  of  ensemble  forecast  data 

•  Developed  code  to  analyze  data  source  performance  characteristics,  for  ensemble  calibration 
and  ambiguity  calculation  training  purposes 

Task  2.  WIP  calculation 

•  Produced  forecast  for  aircraft  icing  using  a  derived  “icing  index”  variable  and  for  aircraft 
turbulence  using  a  derived  “turbulence  index”  variable 

•  Performed  calibration  of  ensemble  forecast  data  for  both  Gaussian  and  non-Gaussian  variables 

•  Performed  flight  route  interpolation 

•  Implemented  "stretching"  of  ensemble  distributions,  separately  at  each  vertical  data  level 
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•  Generated  sample  probability  forecasts 

•  Designed  sample  Mission  Impact  Function  (MIF)  for  potential  effect  of  both  icing  and  turbulence 
on  Army  aviation 

•  Implemented  technique  for  estimation  of  ambiguity  in  probability  forecast  and  in  WIP  calculation 

•  Calculated  WIP  for  individual  variables  (turbulence,  icing)  as  well  as  “total  WIP”  (i.e.,  joint 
probability  of  weather  impact  from  both  turbulence  and  icing) 

Task  3.  Software  architecture  and  design 

•  Architected  and  designed  back-end  server  for  generation  of  WIP  data 

•  Architected  and  designed  front-end  client  for  retrieval,  processing  and  display  of  WIP  data  in 
mission  context 

•  Designed  client/server  interface 

•  Defined  data  file  specification 

Task  4.  User  interface  design 

•  Designed  WIPCast  prototype  user  interface  to  support  the  mission  planner  use  case 

•  Designed  integrated  graphical  and  tabular  displays  of  WIP  data 

•  Designed  intuitive  integrated  displays  for  graphical  representation  of  anticipated  adverse  weather 
impacts  and  risks  at  a  grid  point,  along  a  flight  leg,  and  along  an  entire  flight  path 

•  Designed  controls  for  setting  user  preferences  and  mission  parameters 

Task  5.  Proof -of-concept  prototyping 

•  Developed  a  preliminary  proof-of-concept  working  prototype,  capable  of: 
o  Generating  WIPCast  data  from  probabilistic  forecasts 

o  Using  WIPCast  data  to  drive  a  decision  support  tool  for  the  mission  planner 

•  Prepared  test  dataset  for  use  with  prototype 

•  Extracted  interpolated  data  along  sample  flight  route 

Task  6.  Project  management 

3.4  Phase  I  Open  Issues 

The  following  open  issues  have  emerged  from  the  Phase  I  effort  as  areas  requiring  for  research  in 

Phase  II  (see  Section  5.2.1): 

•  Turbulence  Forecast.  The  WIPCast  approach  to  producing  turbulence  forecasts  employed  in 
Phase  I  produced  good  results  at  upper  levels  (>1 0,000ft)  but  often  considerably  over-forecasted 
turbulence  in  the  low  levels.  An  alternate  approach  that  is  expected  to  effectively  address  this 
issue  has  been  identified  and  needs  to  be  implemented  in  Phase  II. 

•  Representation  of  Ground  Truth.  Current  use  in  WIPCast  of  a  short-range  model  forecast  for 
representation  of  ground  truth  has  serious  limitations  in  realism  and  can  present  a  biased  view. 
An  independent,  unbiased  (or  at  least  only  slightly  biased)  ground  truth  with  a  sufficiently  fine 
spatio-temporal  resolution  covering  the  entire  forecast  domain  is  needed. 
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•  Ambiguity  Calculation.  WIPCast  needs  to  improve  both  accuracy  and  efficiency  of  ambiguity 
calculation.  An  alternate  approach  that  is  expected  to  effectively  address  this  issue  has  been 
identified  and  needs  to  be  implemented  in  Phase  II. 

•  Calibration.  The  validity  and  skill  of  the  calibration  needs  to  be  analyzed.  While  the  shift-and- 
stretch  method  has  been  shown  to  be  effective  in  general  (Eckel  and  Allen  2011),  it  is  unclear 
how  it  works  for  the  derived  variables  used  in  WIPCast,  particularly  with  respect  to  the  adaptation 
to  non-Gaussian  variables. 

•  Weighting  Schemes.  It  is  important  for  WIPCast  to  accommodate  the  ability  to  specify  relative 
weights  of  “importance”  for  individual  weather  sensitivities,  especially  since  relative  importance  of 
individual  weather  events  is  likely  to  vary  based  on  type  of  mission  and  other  operational 
constraints.  A  particularly  interesting  outcome  of  our  Phase  I  research  was  the  realization  that 
user  input  in  defining  impact  thresholds  (as  input  to  the  MIF)  serves  as  an  implicitly  effective 
means  for  relative  weighting  of  weather  sensitivities,  which  is  the  approach  we  therefore  plan  to 
pursue  in  Phase  II. 

•  Decision  Support.  The  Phase  II  effort  will  extend  the  innovative  user  interface  paradigms 
developed  in  Phase  I  to  further  enhance  the  decision  support  capabilities  provided  by  the 
WIPCast  client  display. 

•  3D  Visualization.  One  of  the  visualization  approaches  explored  in  Phase  I  was  that  of  a  3D  cube 
of  the  entire  region  encompassing  the  mission.  Although  such  a  display  would  appear 
conceptually  to  be  beneficial,  we  found  it  particularly  challenging  in  Phase  I  to  devise  a  3D 
visualization  and  interface  paradigm  that  was  sufficiently  intuitive  and  useful  so  as  to  add  value 
beyond  our  multiple  integrated  2D  displays.  We  nevertheless  remain  hopeful  that  some  form  of 
3D  display  will  in  fact  prove  useful  as  part  of  the  WIPCast  user  interface  and  therefore  intend  to 
explore  this  area  further  in  Phase  II. 

3.5  Phase  I  Team 

The  Phase  I  effort  was  spearheaded  by  Impact  Computing,  with  Hyam  Singer  serving  as  Principal 
Investigator,  supported  primarily  by  Mr.  Zev  Hochberg.  Extensive  support  was  provided  by  the 
University  of  Washington’s  Department  of  Atmospheric  Sciences,  under  the  leadership  of 
Dr.  Cliff  Mass,  supported  by  Mr.  Rick  Steed  and  Mr.  Jeff  Baars. 

The  seminal  work  of  Dr.  Tony  Eckel  in  the  domain  of  probabilistic  forecasting  and  its  practical 
application  has  served  as  the  foundation  for  much  of  our  WIPCast  solution.  Our  team  has 
therefore  benefited  greatly  from  the  close  collaboration  and  guidance  of  Dr.  Eckel,  who  has 
provided  invaluable  input  and  feedback  throughout  our  Phase  I  effort. 

Brief  bios  of  the  active  Phase  I  team  members  are  provided  below. 

Mr.  Hyam  Singer,  Impact  Computing  Corp.,  Principal  Investigator 

Mr.  Singer,  Impact’s  President  and  founder,  has  over  30  years  of  experience  in  the  software  industry  and 
has  successfully  managed  the  development  of  a  wide  array  of  innovative,  operationally-relevant  solutions 
for  government  and  commercial  customers.  Mr.  Singer’s  broad  range  of  technical  expertise  spans  the 
areas  of  visualization,  user  interface  design,  geographic  information  systems  (GIS),  image  exploitation, 
modeling  and  simulation  (M&S),  data  mining,  and  complex  systems  architecture.  Mr.  Singer  possesses  a 
proven  track  record  as  a  creative  and  results-oriented  professional,  highly  skilled  at  spearheading 
innovative  solution  development,  from  conceptualization  through  deployment  in  the  field.  Over  the  course 
of  his  career,  Mr.  Singer  has  served  as  Principal  Investigator  for  a  number  of  highly  successful  Phase  1 
and  Phase  2  SBIR  initiatives,  including  a  prior  SBIR  effort  for  ARL.  Mr.  Singer  possesses  a  B.S.  in 
Computer  Science  from  the  University  of  Maryland  and  an  active  Top  Secret  security  clearance. 


29-Jun-2011,  vl.O 


Impact  Computing  Corporation 


Page  23 


SBIR  Topic  A1 0-065  /  Probabilistic  Forecasting  for  Aviation  Decision  Aid  Applications 

Phase  I  Final  Report 


Mr.  Zev  Hochberg,  Impact  Computing  Corp.,  Senior  Systems  Analyst 

Mr.  Hochberg  is  a  technical  innovator  with  over  25  years  of  progressive  technical  and  leadership 
experience  in  software  engineering.  During  his  tenure  at  Impact,  Mr.  Hochberg  has  served  as  lead 
developer  on  a  number  of  SBIR  initiatives  in  addition  to  WIPCast ,  including  a  wiki-based  collaborative 
analysis  and  information  sharing  framework  for  complex  datasets,  and  a  spatio-temporal  data  model  that 
captures  and  represents  the  dynamics  inherent  in  spatio-temporal  datasets  over  both  space  and  time. 

Mr.  Hochberg  has  also  served  as  a  key  member  of  Impact’s  extensible  Behavioral  Modeling  (XBM) 
development  team,  responsible  for  developing  major  portions  of  this  flexible  open  source  framework  for 
integrating  a  hybrid  combination  of  modeling  and  simulation  tools.  Mr.  Hochberg  possesses  an  M.S.  in 
Computer  Science  from  New  York  University’s  Courant  Institute  as  well  as  an  M.A.  in  Physics  from 
Yeshiva  University’s  Belfer  Graduate  School  of  Science. 

Dr.  Clifford  Mass,  University  of  Washington,  Mesoscale  Modeling  Expert 

Dr.  Mass,  Professor  of  Atmospheric  Sciences  at  the  University  of  Washington  (UW)  and  Lead  Scientist  of 
the  Northwest  Modeling  Consortium,  possesses  over  20  years  of  experience  in  mesoscale  prediction,  and 
has  run  a  real-time  mesoscale  ensemble  prediction  effort  for  over  10  years.  With  colleagues  in  the  UW 
statistics  and  psychology  departments,  he  has  built  an  end-to-end  mesoscale  prediction  system  including 
sophisticated  post-processing  and  displays.  He  was  a  member  of  the  NRC  Committee  that  wrote  the 
report  on  the  need  for  uncertainty  information  in  weather  prediction.  Dr.  Mass  possesses  a  Ph.D.  in 
Atmospheric  Sciences  from  the  University  of  Washington.  Among  his  numerous  refereed  publications,  a 
sampling  of  a  few  most  relevant  to  this  initiative  includes: 

Kleiber,  W.,  A.  E.  Raftery,  J.  Baars,  T.  Gneiting,  C.  F.  Mass,  and  E.  Grimit,  2010:  Locally  Calibrated 
Probabilistic  Temperature  Forecasting  Using  Geostatistical  Model  Averaging  and  Local  Bayesian  Model 
Averaging.  Submitted  to  Mon.  Wea.  Rev. 

Mass,  C.,  S.  Joslyn,  J.  Pyle,  P.  Tewson,  T.  Gneiting,  A.  Raftery,  J.  Baars,  J.  M.  Sloughter,  D.  Jones  and 
C.  Fraley,  2009:  PROBCAST:  A  Web-Based  Portal  to  Mesoscale  Probabilistic  Forecasts.  Bulletin  of  the 
American  Meteorological  Society,  90,  1009-1014 

Grimit,  E.  P.,  and  C.  F.  Mass,  2007:  Measuring  the  Ensemble  Spread-Error  Relationship  with  a 
Probabilistic  Approach:  Stochastic  Ensemble  Results.  Monthly  Weather  Review, 135,  203-221 

Eckel,  F.  A.  and  C.  F.  Mass,  2005:  Effective  mesoscale,  short-range  ensemble  forecasting.  Weather  and 
Forecasting,  20,  3238-350 

Mr.  Rick  Steed,  University  of  Washington,  Research  Meteorologist 

Mr.  Steed  possesses  15  years  of  experience  in  mesoscale  numerical  weather  prediction.  For  the  past  10 
years  he  has  constructed  and  maintained  the  UW  mesoscale  ensemble  prediction  system.  During  this 
time  he  participated  in  numerous  projects  which  developed  applications  of  ensemble  weather  forecast 
data  to  probabilistic  decision-making  tools.  Mr.  Steed  possesses  an  M.S.  in  Atmospheric  Sciences  from 
the  University  of  Washington. 

Mr.  Jeff  Baars,  University  of  Washington,  Research  Meteorologist 

Mr,  Baars,  Research  Meteorologist  at  the  University  of  Washington,  possesses  over  15  years  of 
experience,  including  the  last  nine  years  working  with  Dr.  Mass  on  numerical  weather  prediction, 
verification,  and  post-processing.  Mr.  Baars  played  a  key  role  in  the  creation  of  Probcast,  a  pioneering 
probabilistic  weather  prediction  system,  as  well  as  enhancing  and  implementing  Probcast  algorithms  on 
the  AFWA  Joint  Ensemble  Forecast  System.  Mr.  Baars  possesses  an  M.S.  in  Atmospheric  Sciences 
from  the  Ohio  State  University. 
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4  Phase  I  Option 

Our  intent  is  to  use  the  Phase  I  Option  Period  to  refine  the  Phase  I  results  so  as  to  further  set  the 
foundation  for  Phase  II.  Likely  focus  areas  for  the  Phase  I  Option  Period  activities  will  include: 

•  Multivariate  calculations  and  analysis 

•  Additional  visualization  techniques  and  enhancements  to  the  user  interface 

•  Availability  and  use  of  AFWA  (Armed  Forces  Weather  Agency)  data 

•  WIPCast  validation  and  verification  (V&V)  techniques,  including  source  of  “ground  truth”  data 

•  Web  (i.e.,  browser-based)  client 

•  GoogleEarth  integration 

Final  selection  of  Phase  I  Option  Period  activities  will  be  based  on  customer  input  and  prioritization. 
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5  Phase  II  Plan 

We  have  carefully  designed  and  implemented  the  Phase  I  working  prototype  of  our  WIPCast  solution  to 
serve  as  a  firm  technical  foundation  for  a  subsequent  Phase  II  effort.  In  addition,  our  Phase  I  research  - 
along  with  feedback  and  guidance  from  the  customer  -  has  yielded  insights  highly  relevant  to  design  and 
development  of  the  Phase  II  WIPCast  solution. 

The  ultimate  objective  of  the  proposed  follow-on  Phase  II  effort  is  to  develop  an  operational  version  of  the 
WIPCast  system,  building  upon  our  successful  design  and  prototyping  activities  conducted  during 
Phase  I.  A  functional  overview  of  our  vision  for  the  Phase  II  system  is  depicted  in  Figure  14.  . 


Figure  14.  WIPCast  Phase  II  Functional  Overview 

5. 1  Phase  II  Plan  Overview 

The  following  is  a  high-level  overview  of  our  Phase  II  plan: 

•  Goal:  Fully  operational  “version  1 .0”  of  the  WIPCast  system 

•  Period  of  performance:  24  months 

•  Deliverables:  WIPCast  1.0  software  (open  source),  Technical  Documentation, 

Monthly  Status  Reports,  Final  Report 

•  Schedule:  Iterative  development,  with  interim  customer  meetings  and  deliverables  every  6 
months 

•  Team:  All  members  of  Phase  1  team,  expanded  to  include  ZedX  (weather  data  processing  and 
modeling)  and  Marcus  Weather  (commercialization) 

5. 2  Phase  II  Technical  Objectives 

Specific  Phase  II  technical  objectives,  as  detailed  in  the  subsections  that  follow,  consist  of  addressing 
Phase  I  open  issues,  addressing  key  requirements  of  the  operational  WIPCast  system,  and  laying  the 
groundwork  for  WIPCast  productization. 

5.2.1  Addressing  Phase  I  Open  Issues 

A  key  goal  of  the  Phase  II  effort  will  be  to  further  research  and  effectively  address  the  following  open 
issues  (previously  detailed  in  Section  3.4)  that  emerged  from  the  Phase  I  effort: 

•  Turbulence  Forecast.  See  Section  5.3.9. 

•  Representation  of  Ground  Truth.  See  Section  5.3.8. 

•  Ambiguity  Calculation.  See  Section  5.3.4. 

•  Calibration.  See  Section  5.3.5. 

•  Weighting  Schemes.  See  Section  5.3.7. 

•  Decision  Support.  See  Section  5.3.10. 

•  3D  Visualization.  See  Section  5.3.10. 
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5.2.2  Key  Operational  Requirements 

In  addition  to  addressing  the  open  technical  issues  enumerated  in  Section  5.2.1 ,  there  are  a  number  of 
key  operational  requirements  that  must  be  satisfied  to  achieve  a  version  of  the  WIPCast  system  that  is 
sufficiently  marketable  and  deployable,  including: 

•  Support  for  additional  weather  variables.  While  the  focus  on  turbulence  and  icing  was  entirely 
adequate  for  our  Phase  I  proof-of-concept  prototyping  efforts,  the  Phase  II  system  will  need  to 
incorporate  support  for  additional  weather  variables  of  interest.  See  Section  5.3.1 . 

•  Dynamic  WIP  Calculation.  WIPCast  needs  to  build  the  capability  to  recompute  output  on-the-fly 
based  on  dynamic  user  or  external  inputs  regarding  flight  route  (path,  time,  altitude),  weather 
sensitivity  impact  thresholds  (for  defining  the  MIF),  and  risk  tolerance.  See  Section  5.3.2. 

•  Multivariate  WIP  Calculation.  The  chance  of  a  negative  impact  depends  on  multiple  elements, 
such  as  wind  speed  and  direction,  visibility,  turbulence,  and  icing.  In  addition,  it  depends  on 
these  quantities  at  multiple  points  in  time  and  space  across  the  mission.  Thus,  in  practice,  the 
total  WIP  depends  on  a  large  number  of  inter-related  weather  phenomena  simultaneously  and 
therefore  necessitates  the  ability  to  address  the  multivariate  WIP  challenge.  See  Section  5.3.3. 

•  Integration  with  External  Systems  and  Data  Sources.  Formal  definition  of  the  WIPCast 
Server  APIs  (Application  Programming  Interfaces)  in  Phase  II  will  facilitate  integration  with  other 
systems  such  as  MyWIDA,  T-IWEDA,  JMPS,  and  AWRT.  These  systems  will  thereby  be  able  to 
exploit  outputs  of  the  WIPCast  system,  as  well  as  providing  mission  configuration  and  other 
inputs  to  WIPCast.  In  addition,  the  WIPCast  front-end  geospatial  display,  which  was  integrated 
with  GoogleMaps  in  Phase  I,  will  also  be  integrated  with  GoogleEarth  in  Phase  II.  See  Section 
5.3.11. 

•  Browser-based  Client.  The  Phase  I  prototype  provided  a  desktop  client.  Also  providing  a  web 
browser-based  client  as  an  alternative  provides  important  operational  benefits,  such  as  ease  of 
deployment  and  accessibility.  See  Section  5.3.10. 

5.2.3  Productization 

With  an  eye  toward  productization  and  commercialization,  the  Phase  II  effort  is  also  to  yield: 

•  Comprehensive  user  and  technical  documentation 

•  Suite  of  automated  functional  and  regression  test  scripts 

•  Verification  and  validation  (V&V)  plan 
Further  detail  is  provided  in  Sections  5.3.12  and  5.3.13. 


5. 3  Phase  II  Tasks  and  Research  Areas 

The  specific  tasks  to  be  performed  and  research  areas  to  be  addressed  in  Phase  II  are  detailed  in  the 
subsections  that  follow.  Each  task  directly  supports  one  or  more  of  the  technical  objectives  listed  in 
Section  5.2. 

5.3.1  Support  for  Additional  Weather  Variables 

While  the  focus  on  turbulence  and  icing  was  adequate  for  our  Phase  I  proof-of-concept  prototyping 
efforts,  the  Phase  II  system  will  need  to  incorporate  support  for  additional  weather  variables  of  interest. 
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In  support  of  the  Army  aviation  use  case,  a  Phase  II  test  case  will  most  likely  be  selected  that 
focuses  on  a  specific  aircraft  (e.g.,  Apache)  and  extends  WIPCast  capabilities  beyond  icing  and 
turbulence  to  also  supporting  at  least  3  additional  weather  parameters  of  interest  (such  as  cloud 
ceiling,  visibility,  winds,  precipitation,  etc.). 


5.3.2  Dynamic  WIP  Calculation 

WIPCast  needs  to  build  the  capability  to  recompute  output  on-the-fly  based  on  dynamic  user  or  external 
inputs  regarding  flight  route  (path,  time,  altitude),  weather  sensitivity  impact  thresholds  (for  defining  the 
MIF),  and  risk  tolerance.  For  a  change  only  in  risk  tolerance,  WIPCast  will  just  update  the  R-Y-G  decision 
recommendations  along  the  route.  For  a  change  in  impact  thresholds,  WIPCast  will  update  the  MIF  so  as 
to  recompute  WIP  and  its  90%  Cl  (using  RCR)  along  the  flight  route.  For  a  change  in  flight  route, 
WIPCast  will  have  to  recompute  everything  beginning  with  interpolation  of  the  original  ensemble  forecast 
data  along  the  route. 

5.3.3  Multivariate  WIP  Calculation 

Meeting  the  challenge  of  accurately  calculating  a  total  WIP  that  accounts  for  concurrent  risks  from 
multiple  weather  sensitivities  has  been  deferred  to  Phase  II.  For  Phase  II,  we  plan  to  first  develop  an 
accurate  method  for  calculating  total  (i.e.,  multivariate)  WIP  starting  with  the  relatively  simply  two-event 
(i.e.,  icing  and  turbulence)  problem.  After  building  a  foundation  with  the  two-event  problem,  we  will 
expand  to  handling  more  variables  (slant  range  visibility,  cloud  ceiling,  wind  speed,  etc.)  simultaneously. 

In  the  Phase  I  prototype,  WIP  was  generated  separately  for  two  weather  sensitivities  (turbulence  and 
icing)  and  a  total  WIP  was  produced  that  accounts  for  both  risks  using  a  simplification.  To  understand  the 
simplification,  as  well  as  the  larger  challenge,  we  present  the  following  basic  explanation. 

First,  consider  just  the  two  event  (i.e.,  icing  and  turbulence)  problem.  We  define  A  as  being  an  impact 
from  TURBC,  so  Pr(A)  is  WIP  from  TURBC,  and  B  as  being  an  impact  from  ICG,  so  Pr(B)  is  WIP  from 
ICG.  The  total  WIP  is  then  the  chance  of  either  impact  (A  or  B)  occurring: 

Pr(AuB)  =  Pr(A)  +  Pr(B)  -  Pr(AnB) 

Calculating  the  last  term  on  the  right  -  Pr(AnB)  -  is  where  the  challenge  lies.  It  is  known  as  the  joint 
probability  and  is  best  understood  with  a  Venn  diagram: 


Figure  15.  Venn  diagram  example  of  two-event  situation  with  Pr(A)  =  0.4  and  Pr(B)  =  0.3 

The  joint  probability  is  the  overlap  between  the  two  events’  probabilities,  within  the  space  of  all  possible 
outcomes  (i.e.,  the  complete  box).  Total  WIP  is  the  fraction  of  the  box  that  is  shaded  any  color.  So  as  to 
avoid  counting  the  overlap  area  (i.e.,  the  green  area)  twice,  the  equation  above  subtracts  the  overlap  from 
the  total  of  the  two  event  probabilities.  However,  calculating  the  actual  value  of  the  joint  probability  is 
anything  but  simple: 


or  equivalently: 


Pr(AnB)  =  Pr(A)*Pr(B|A) 

Pr(AnB)  =  Pr(B)*Pr(A|B) 
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The  conditional  probability,  PrA|B),  is  normally  not  known  and  difficult  to  estimate  when  the  two  events 
have  a  variable  degree  of  correlation  (i.e.,  are  not  independent).  This  is  the  case  for  TURBC  and  ICG 
(the  two  variables  used  in  the  Phase  I  prototype)  since  they  often  go  hand-in-hand  (e.g.,  a  stronger  storm 
has  both  more  intense  TURBC  and  ICG),  but  can  also  correlate  weakly  at  other  times.  Note  that  the 
correlation  of  interest  here  is  on  occurrence  of  an  event  (i.e.,  binary  Y/N  for  impact  from  TURBC  or  ICG) 
and  not  correlation  between  two  continuous  random  variables. 

Substituting  in  the  expression  for  joint  probability  to  the  total  WIP  equation,  we  simplify  to: 

Pr(AuB)  =  Pr(A)  +  Pr(B)  *  [1  -  Pr(A|B)] 

This  provides  a  logical  perspective  for  the  challenge.  The  total  WIP  is  found  by  starting  with  the  larger  of 
the  two  WIPs  and  adding  on  part  of  the  smaller  WIP,  so  the  minimum  total  WIP  is  Pr(A).  The  fractional 
part  of  Pr(B)  to  add  on  is  related  to  the  correlation  between  A  and  B,  as  seen  in  Figure  16.  . 


Anti-Correlated  No  Correlation  Correlated 


p-^,ormin?< - >  p-  0  < - >p-1,ormax? 

(max  exclusivity )  (independenf)  (max  inclusivity) 


— -xFr(AnB) 

(  Pr  (Af)\ 

(Pr(B)=  1 
\\Pr(AnBy 

Pr(A|B)  =  0.0  Pr(A|B)  =  Pr(A)  Pr(A|B)  =  1.0 

Pr(B)*[1  -  Pr(A|B)]  =  0.3  Pr(B)*[1  -  Pr(A|B)j  =0.18  Pr(B)*[1  -  Pr(A|B)]  =  0.0 

[  100%  ]  < - ■»  [  60%  ]  < - ->[0%] 

Pr(AuB)  =  0.4  +  0.3  =  0.7  Pr(AuB)  =  0.4  +  0.18  =  0.58  Pr(AuB)  =  0.4  +  0.0  =  0.4 

Figure  16.  Depiction  of  total  WIP  calculation,  Pr(AuB),  over  the  continuum  of  possible  amounts  of 
correlation  (indicated  by  the  correlation  coefficient,  p)  between  A  and  B.  The  blue  %  values  are 
the  fraction  of  Pr(B)  that  must  be  added  to  Pr(A)  to  arrive  at  the  total  WIP. 

In  the  anti-correlated  case,  when  both  probabilities  are  small  enough,  they  can  be  mutually  exclusive  and 
have  p  =  -1  as  shown  here.  So  for  total  WIP,  all  of  Pr(B)  gets  added  on  to  Pr(A).  Once  the  sum  of  the 
two  probabilities  is  >  1.0,  there  will  be  some  overlap,  both  Pr(A|B)  and  yo  will  be  at  some  minimum  value, 
and  Pr(AuB)  =  1.0. 

On  the  opposite  extreme,  when  A  and  B  are  highly  correlated,  they  can  be  perfectly  correlated  only  if  they 
are  equal,  otherwise  p  will  be  at  some  maximum  value.  However  Pr(A|B)  always  equals  1 .0  since  we 
chose  the  perspective  that  Pr(A)  >  Pr(B).  So  there  is  nothing  to  add  onto  to  Pr(A)  from  Pr(B)  and  total 
WIP  is  simply  Pr(A). 

In  the  middle  case,  A  and  B  occur  independently  but  can  still  occur  at  the  same  time.  The  joint  probability 
simplifies  to  Pr(A)*Pr(B)  and  the  conditional  probabilities  are  equal  to  the  probability  of  the  second  event. 
This  is  the  case  we  assumed  in  calculating  total  WIP  for  Phase  I. 

To  emphasize  the  importance  of  risk  analysis  with  total  WIP,  consider  a  decision  by  a  user  with  a  risk 
tolerance  of  45%  who  is  given  the  risks  depicted  in  Figure  1 6. .  Risk  analysis  performed  separately  on 
each  of  the  two  possible  events  would  result  in  an  overall  ‘Go’  decision  since  risk  is  acceptable  from  both 
TURBC  and  ICG  individually.  However,  research  shows  that  p  =  0.2  at  these  levels  of  risk,  so  total  WIP  = 
0.54  and  the  appropriate  decision  is  ‘No-Go’  since  there  is  too  much  risk. 

For  Phase  II,  we  plan  to  further  explore  the  continuum  depicted  in  Figure  16.  to  develop  an  accurate 
method  of  calculating  total  WIP,  starting  with  the  relatively  simply  two-event  problem.  The  basic  approach 
is  to  find  either  the  correlation  between  the  two  events  and  relate  it  to  the  joint  probability,  or  find  the 
conditional  probability  directly.  Note,  however,  that  this  involves  exploring  the  events  of  TURBC  or  ICG 
(at  or  above  a  selected  intensity  level)  rather  than  the  true  end  goal  of  exploring  the  events  of  an  impact 
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from  TURBC  or  ICG.  Since  we  are  unable  to  track  the  event  of  actual  impacts  from  TURBC  or  ICG,  we 
will  track  occurrence  of  TURBC  and  ICG,  assume  an  impact,  and  transfer  the  resulting  relationships  to 
apply  to  WIP. 

There  are  many  possible  dependencies  to  explore  for  possible  ways  to  condition  the  variability  in  the 
correlation  between  the  two  events.  A  prominent  one  is  simply  that  the  correlation  depends  on  the 
amount  of  probability  itself.  Large  probability  values  naturally  must  carry  high  correlation,  but  low 
probability  values  may  go  either  way  depending  upon  the  variables  in  question.  Another  dependency  to 
examine  is  the  weather  regime.  The  correlation  between  the  events  may  change  dramatically  over 
different  seasons,  atmospheric  stability,  topography,  etc.  Lastly,  the  relative  intensity  of  the  event  may 
play  a  role  since  light-moderate  conditions  could  conceivably  vary  independently  while  severe  conditions 
go  more  hand-in-hand. 

After  building  a  foundation  with  the  two-event  problem,  we  will  expand  to  handling  more  variables  (slant 
range  visibility,  cloud  ceiling,  wind  speed,  etc.)  simultaneously.  While  we  recognize  that  the  complexity 
increases  substantially  with  more  variables  (since  it  involves  3-way,  4-way,  etc.  correlations  or  conditional 
probabilities),  the  risk-analysis  decision  still  ultimately  comes  down  to  comparing  the  user’s  risk  tolerance 
to  the  best  possible  approximated  estimate  of  the  total  WIP. 

5.3.4  Improved  Ambiguity  Calculation 

WIPCast  needs  to  improve  both  accuracy  and  efficiency  of  ambiguity  calculation.  Toward  that  end,  in 
Phase  II: 

•  The  RCR  code  will  be  completed  to  include  the  random  calibration  component. 

•  As  a  huge  boost  to  efficiency,  the  RCR  code  will  be  adapted  to  run  only  along  the  flight  route 
(which  will  require  interpolation  of  all  ensemble  member  data  along  the  route). 

•  The  higher-performance  CES  method  for  ambiguity  calculation  will  then  be  implemented  to 
generate  90%  confidence  intervals  (Cl)  for  forecast  probability  and  WIP  over  the  complete 
domain. 

While  there  will  be  some  disagreement  between  the  RCR  and  CES  results,  WIPCast  will  employ  CES  to 
convey  a  general,  overall  view  of  the  ambiguity  while  RCR  will  be  used  to  present  a  more  refined  view 
where  it  is  needed  (e.g.,  along  the  flight  route). 

5.3.5  Calibration 

The  validity  and  skill  of  the  calibration  needs  to  be  analyzed.  While  the  shift-and-stretch  method  has 
been  shown  to  be  effective  in  general  (Eckel  and  Allen  2011),  it  is  unclear  how  it  works  for  the  derived 
variables  used  in  WIPCast,  particularly  with  respect  to  the  adaptation  to  non-Gaussian  variables. 

A  thorough  verification  analysis  is  to  be  performed  in  Phase  II,  the  results  of  which  will  likely  reveal 
multiple  alternatives  for  further  improving  the  calibration,  such  as  conditioning  the  2nd  moment  correction 
on  the  ensemble  spread  (Eckel  and  Allen  2011). 

5.3.6  MIF  Specification  and  Weighting  of  Weather  Sensitivities 

One  approach  for  defining  a  MIF  is  to  allow  for  input  from  the  users.  The  idea  is  that  the  user  has 
knowledge  of  the  factors  that  can  affect  the  likelihood  of  impact  (e.g.,  flight  crew  experience,  aircraft  load, 
aircraft  age,  etc.)  for  a  given  observed  weather  condition,  and  should  thus  adjust  the  MIF  accordingly. 

For  example,  a  very  experienced  pilot  may  be  able  to  handle  MDT  TURBC  much  better  than  a  novice 
pilot  and  thus  greatly  increase  the  odds  of  mission  success  when  MDT  TURBC  occurs.  The  user  would 
then  be  justified  in  adjusting  the  corresponding  TURBC  MIF  threshold  to  be  higher. 

Varying  the  impact  thresholds,  and  thereby  the  MIF,  can  greatly  change  the  final  value  of  WIP,  perhaps 
more  accurately  reflecting  the  overall  risk  to  the  mission.  Rather  than  dealing  with  a  complex  CDF, 
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WIPCast  will  allow  a  user  to  specify  the  weather  level  at  which  there  is  a  minor  chance  of  impact  and  a 
major  chance  of  impact  (Figure  17. ),  based  on  their  knowledge  of  factors  such  as  the  skill  and  fatigue  of 
the  team,  the  age  of  the  equipment,  and  so  on.  There  are  then  assumed  to  be  the  10th  and  90th 
percentiles  of  the  MIF  which  are  calculated  behinds  the  scenes.  The  MIF  thereby  represents  the 
uncertainty  in  an  impact  from  actual  weather  due  to  all  remaining  unclear  factors. 


Further  research  with  regard  to  how  to  appropriately  define  a  MIF  for  each  weather  sensitivity  is  also 
planned  for  Phase  II. 


Select  Impact  Thresholds 


FCfidc  and  drso  a  tiveshold  marker  Si  adjust  it! 

TURBULENCE 


TURB  MIF  for  Rotary-Wing  Aircraft 


Figure  17.  User  interface  for  selecting  impact  thresholds.  Top  left  image  shows  default  settings  for 
turbulence  impact  thresholds.  Lower  right  image  shows  sample  adjustment  by  user,  with  the 
resulting  MIF  shown  in  the  graph  on  the  right. 


5.3.7  Weighting  Schemes 

For  any  given  mission,  a  variety  of  factors  may  cause  certain  events  to  be  more  important  or  significant  to 
mission  success  than  others.  Accordingly,  it  is  important  for  WIPCast  to  accommodate  the  ability  to 
specify  relative  weights  of  “importance”  for  events.  However,  as  discussed  in  our  Phase  I  proposal,  there 
are  significant  obstacles  to  defining  a  weighting  scheme  that  has  anywhere  near  universal  applicability  or 
that  will  be  equally  agreeable  to  a  large  population  of  users.  In  particular: 

•  Relative  importance  of  individual  events  is  likely  to  vary  based  on  type  of  mission  and  other 
operational  constraints 

•  Given  the  interdependence  of  multiple  events,  a  simple  relative  weighting  scheme  may  not  be 
adequate  to  address  the  complexities  of  event  prioritization 

•  Individual  users  may  not  agree  on  the  appropriate  relative  weights  to  apply  to  relevant  events, 
even  in  the  same  operational  scenario 

A  particularly  interesting  outcome  of  our  Phase  I  research  was  the  realization  that  user  input  in 
defining  impact  thresholds  (as  input  to  the  MIF)  serves  as  an  implicitly  effective  means  for  relative 
weighting  of  weather  sensitivities,  (e.g.,  risk  from  turbulence,  lightening,  low  visibility,  etc.).  The  higher 
the  sensitivity  to  a  particular  weather  condition,  the  lower  the  corresponding  impact  threshold  will  be  set 
by  the  user  (since  the  threshold  determines  the  point  beyond  which  a  negative  impact  is  anticipated),  and 
vice  versa.  For  example,  if  a  user  is  more  concerned  about  the  potential  impact  from  icing  than  the 
impact  from  turbulence,  he/she  would  then  adjust  the  thresholds  accordingly  (i.e.,  lowering  the  impact 
threshold  for  icing  to  indicate  greater  sensitivity  and/or  raising  the  impact  threshold  for  turbulence  to 
indicate  lower  sensitivity),  thereby  implicitly  giving  icing  heavier  weighting  than  turbulence  in  total  WIP 
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calculation.  These  implied  relative  weights  will  then  carry  over  into  the  calculation  of  total  WIP,  which  will 
thereby  reflect  a  weighted,  total  chance  of  impact. 

5.3.8  Weather  Observation  Data  (“Ground  Truth”) 

Current  use  in  WIPCast  of  a  short  range  model  forecast  for  representation  of  ground  truth  has  serious 
limitations  in  realism  and  can  present  a  biased  view.  An  independent,  unbiased  (or  at  least  only  slightly 
biased)  ground  truth  with  a  sufficiently  fine  spatio-temporal  resolution  covering  the  entire  forecast  domain 
is  needed.  Assuming  a  CONUS  forecast  domain,  the  target  for  surface  variables  is  the  Real  Time 
Mesoscale  Analysis  (RTMA),  which  is  currently  produced  hourly  on  a  5km  grid  and  will  soon  be  upgraded 
to  a  2.5km  grid.  RTMA  data  is  available  for  free  from  the  NWS.  For  non-surface  data,  the  target  is  the 
Rapid  Update  Cycle  (RUC)  analysis,  which  is  produced  hourly  on  a  13km  grid  covering  CONUS.  RUC13 
data  is  available  from  NOAA. 

5.3.9  Improved  Turbulence  Forecast 

The  turbulence  index  (77)  algorithm  employed  in  Phase  I  followed  the  method  presented  by  Ellrod  and 
Knapp  (1992),  which  considers  the  wind  shear,  deformation  and  convergence  in  the  forecasted,  three- 
dimensional  model  wind  field.  Given  the  UWME  model  resolution  of  12km,  we  followed  AFWA’s 
empirically  based  thresholds  for  mesoscale  models  of  LGT  (3  <  77  <  9),  MDT  (9  <  77  <  14),  and  SVR  (14  < 
77).  This  produced  good  results  at  upper  levels  (>1 0,000ft)  but  often  considerably  over-forecasted 
turbulence  in  the  low  levels. 

This  issue  can  be  alleviated  by  implementing  the  Panofsky  Index  (PI)  to  represent  turbulence  in  the  lower 
atmosphere  (below  about  5,000ft  AGL)  (Boyle  1990,  Passner  2000,  Brooks  and  Oder  2004).  A  challenge 
will  be  how  to  smoothly  blend  from  PI  to  Tl  to  avoid  any  discontinuities  in  the  final  product.  One 
possibility  is  to  leverage  how  AFWA  deals  with  that  challenge  since  they  use  these  indices  in  operations. 

5.3.1 0  Decision  Support  and  User  Interface 

The  most  powerful  and  well  architected  systems  are  often  rendered  essentially  useless  by  an  ill-designed 
user  interface.  As  Donald  Norman  states  in  his  insightful  book  The  Design  of  Everyday  Things,  "There  is 
a  big  difference  between  the  expertise  required  to  be  a  designer  and  that  required  to  be  a  user."  Norman 
goes  on  to  state  that,  whereas  designers  become  expert  with  the  device  they  are  designing,  users 
become  expert  at  the  task  they  are  trying  to  perform  with  the  device.  These  two  perspectives  are 
unfortunately  often  worlds  apart.  At  Impact  Computing,  our  approach  to  user  interface  design  is  therefore 
highly  "user-centric",  relying  heavily  on  an  iterative  cycle  of  rapid  prototyping  and  user  feedback  to  ensure 
a  solution  that  is  operationally  relevant,  intuitive  and  easy-to-use. 

Accordingly,  the  Phase  I  prototype’s  WIPCast  display  intuitively  and  clearly  presents  anticipated  adverse 
weather  impacts  and  risks  at  a  grid  point,  along  a  flight  leg,  and  along  an  entire  flight  path.  The  interface 
also  provides  a  mechanism  for  the  user  to  characterize  their  risk  tolerance,  which  then  helps  drive  the 
information  presented  in  the  resulting  decision  support  displays. 

Building  on  this  foundation,  Phase  II  research  and  development  tasks  relating  to  the  WIPCast  decision 
support  front-end  will  include: 

•  Creation  of  a  mission  specification  “wizard”  enabling  users  to  easily  input  mission 
configuration  parameters 

•  Additional  2D  displays,  such  as: 

o  A  graphical  representation  of  predicted  weather  forecast  values  as  compared  with 

corresponding  parameter  threshold  values  (this  display  was  suggested  by  the  customer 
at  our  final  Phase  I  meeting  and  would  provide  an  useful  added  dimension  of  information 
complementing  the  existing  WIP  graph  display  developed  in  Phase  I) 
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o  A  dashboard  (i.e.,  “stop  light”)  display  summarizing  the  anticipated  impacts  and  risk 
levels  for  each  of  the  weather  parameters  being  evaluated 

•  3D  cube  of  the  entire  region  encompassing  the  mission5 

•  Additional  spatio-temporal  controls  to  allow  for  further  manipulation  of  the  WIPCast  displays 
over  both  space  and  time 

•  Integration  of  the  WIPCast  map  display  with  Google  Earth,  complementing  the  existing 
Phase  I  integration  with  Google  Maps 

•  Browser-based  client,  as  an  alternative  to  the  existing  desktop  client,  for  added  ease  of 
deployment  and  accessibility 

5.3.1 1  Software  Architecture,  Design  and  System  Integration 

WIPCast  is  based  on  a  Service  Oriented  Architecture  (SOA),  so  as  to  maximize  interoperability  with 
existing  systems  and  frameworks,  such  as  AWRT,  T-IWEDA,  MyWIDA,  DCGS-A,  and  Google  Earth. 

Consistent  with  sound  software  engineering  and  architectural  principals,  the  WIPCast  architecture 
provides  a  clean  and  clear  separation  between  the  data  layer  (i.e.,  the  mesoscale  ensemble  systems  and 
their  output),  the  logical  layer  (i.e.,  WIPCast  Server)  and  the  user  interface  layer  (i.e.,  WIPCast  clients). 

Of  particular  import  is  the  fact  that  the  WIPCast  system  is  being  architected  to  avoid  reliance  on 
the  peculiarities  or  idiosyncrasies  of  any  particular  mesoscale  ensemble  data  source. 

Formal  definition  of  the  WIPCast  Server  APIs  (Application  Programming  Interfaces)  in  Phase  II  will 
facilitate  integration  with  other  systems  such  as  MyWIDA,  T-IWEDA,  JMPS,  and  AWRT.  These  systems 
will  thereby  be  able  to  exploit  outputs  of  the  WIPCast  system,  as  well  as  providing  mission  configuration 
and  other  inputs  to  WIPCast. 

Specific  integration  tasks  of  potential  interest  in  Phase  II  include: 

•  Integration  and  exploitation  of  AFWA  ensemble  data 

•  Acquisition  and  integration  of  NWS  fine  scale  analysis  grids 

•  Integration  with  other  systems  and  services  to  provide  mission  profile  data  (flight  route,  etc.) 

•  MyWIDA  integration 

•  Integration  with  other  tools  (e.g.,  Palantier,  ParaView,  VTK,  etc.) 

In  addition,  as  discussed  in  Section  5.3.10,  the  WIPCast  front-end  geospatial  display,  which  was 
integrated  with  GoogleMaps  in  Phase  I,  will  also  be  integrated  with  GoogleEarth  in  Phase  II. 

5.3.12  Documentation 

A  comprehensive  set  of  user  and  technical  documentation  will  be  developed  as  part  of  the  Phase  II  effort 
including: 

•  User  Manual 

•  System  Architecture  Document 


0  Although  a  3D  display  would  appear  conceptually  to  be  quite  useful,  we  found  it  particularly 
challenging  in  Phase  I  to  devise  a  3D  visualization  and  interface  paradigm  that  was  sufficiently  intuitive 
and  useful  so  as  to  add  value  beyond  our  multiple  integrated  2D  WIPCast  displays.  We  nevertheless 
remain  hopeful  that  some  form  of  3D  display  will  in  fact  prove  useful  as  part  of  the  WIPCast  user  interface 
and  therefore  intend  to  explore  this  area  further  in  Phase  II. 
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•  Complete  set  of  JavaDoc  for  Java  classes 

Monthly  status  reports  shall  also  be  provided  throughout  the  Phase  II  period  of  performance  as  all  as  a 
final  technical  report  upon  completion  of  Phase  II. 

5.3.13  Testing 

A  suite  of  automated  functional  and  regression  test  scripts  will  be  developed,  maintained  and  executed  as 
part  of  the  Phase  II  effort.  JUnit  (http://www.iunit.org/)  will  most  likely  be  used  for  automated  functional 
testing  and  JMeter  (http://iakarta.apache.org/imeter/)  will  most  likely  be  used  for  performance  testing  and 
benchmarking. 

A  Verification  and  Validation  (V&V)  plan  for  the  system  is  also  to  be  developed  as  part  of  the  Phase  II 
effort. 

5.3.14  Additional  Research  Areas  to  Consider 

The  tasks  and  research  areas  detailed  in  the  preceding  sections  are  considered  to  be  the  highest  priority 
next  steps  and  have  therefore  collectively  been  selected  as  the  focus  of  the  Phase  II  effort. 

There  are  undoubtedly  additional  research  areas  worth  pursuing,  either  in  Phase  II  if  time  permits,  or 
subsequently  as  part  of  further  product  development  and  enhancement  efforts. 

Two  particular  research  topics  that  bear  special  consideration  are: 

•  Calculating  “overall  WIP”  for  the  entire  mission.  In  the  context  of  decision  support,  it  would 
be  beneficial  to  provide  the  decision  maker  with  a  “bottom  line”  overall  risk  evaluation.  Research 
will  need  to  be  conducted  to  determine  the  most  effective  way(s)  of  synthesizing  an  overall  WIP 
for  the  mission,  spanning  all  relevant  forecast  data  across  the  mission’s  space-time  continuum 
(e.g.,  across  the  entire  flight  path). 

•  Extending  risk  evaluation  to  incorporate  other  anticipated  impacts.  Using  the  Army  aviation 
use  case  as  an  example,  weather  impact  analysis  should  not  be  limited  to  the  flight  route  when 
airborne  but  also  should  incorporate  terminal  forecasts  since  takeoff  and  landing  are  often  the 
riskiest  parts  of  the  mission. 


5.4  Phase  II  Team 

We  are  pleased  to  propose  a  Phase  II  Project  Team  that  includes  all  individuals  who  played  a  key 
role  in  the  Phase  I  effort,  in  the  same  roles,  thus  providing  the  Army  Research  Lab  (ARL)  with  the 
benefit  of  our  team’s  accumulated,  collective  project  expertise  to  date. 

In  particular,  the  Phase  II  effort  will  again  be  spearheaded  by  Impact  Computing,  with  Hyam  Singer 
serving  as  Principal  Investigator,  supported  primarily  by  Mr.  Zev  Hochberg  and  Dr.  David  Makovoz. 
Extensive  support  will  again  be  provided  by  the  University  of  Washington’s  Department  of 
Atmospheric  Sciences,  under  the  leadership  of  Dr.  Cliff  Mass,  supported  by  Mr.  Rick  Steed  and 
Mr.  Jeff  Baars.  Dr.  Tony  Eckel,  whose  seminal  work  in  the  domain  of  probabilistic  forecasting  and  its 
practical  application  serves  as  the  foundation  for  much  of  our  WIPCast  solution,  is  to  continue  as  a  close 
and  active  advisor  to  our  team  throughout  the  Phase  II  effort. 

In  addition,  the  Phase  II  team  will  be  augmented  by  Marcus  Weather  Inc.  (spearheading 
commercialization  efforts)  and  ZedX  Inc.  (providing  hosting  services  and  development  support).  The 
presidents  of  both  of  these  companies  -  Mr.  Kevin  Marcus  of  Marcus  Weather  and  Dr.  Joe  Russo  of 
ZedX  -  are  to  be  active  members  of  the  Phase  II  team. 
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Brief  bios  of  those  Phase  II  team  members  who  served  actively  on  the  Phase  I  team  are  provided 
in  Section  3.5.  Brief  bios  for  those  additional  individuals  who  are  to  augment  the  Phase  I  team  in 
Phase  II  are  provided  below. 


Dr.  David  Makovoz,  Impact  Computing  Corp.,  System  Architect 

Dr.  Makovoz  possesses  over  25  years  of  experience  in  applied  R&D.  With  a  background  firmly  rooted  in 
physical  science,  Dr.  Makovoz  has  spearheaded  a  number  of  innovative,  leading  edge  initiatives  across  a 
wide  array  of  complex  domains  including  machine  learning,  pattern  recognition,  image  processing,  signal 
processing,  information  retrieval,  data  mining,  modeling  and  simulation,  and  natural  language  processing. 
Dr.  Makovoz  possesses  a  unique  blend  of  experience,  spanning  algorithm  development,  mathematical 
modeling,  statistical  analysis,  complex  system  architecture  and  design,  and  programming  in  a  variety  of 
languages.  He  possesses  expertise  in  high  performance  parallel  computing  and  distributed  programming 
on  heterogeneous  networks.  Dr.  Makovoz  currently  serves  as  Impact  Computing’s  Director  of  Research 
and  Development.  Prior  to  joining  Impact  Computing,  he  served  as  Chief  Engineer  at  Caltech's  Infrared 
Processing  and  Analysis  Center  and  Principal  Analyst  at  CACI.  Dr.  Makovoz  possesses  a  Ph.D.  in 
Physics  from  the  University  of  Washington  as  well  as  an  active  Secret  security  clearance. 

Mr.  Joe  Russo,  ZepX  Inc.,  Senior  Technical  Consultant 

Dr.  Russo  is  a  co-founder  and  president  of  ZedX,  Inc.,  an  information  technology  company  that 
specializes  in  custom  weather  databases,  decision-support  algorithms,  and  data  visualization  tools  for  the 
agricultural  and  environmental  industries.  Since  1999,  Dr.  Russo  has  been  involved  in  the  design  and 
development  of  interactive,  web-based,  decision-support  and  learning  services  for  the  agricultural  sector. 
Since  2002,  Dr.  Russo  has  also  been  involved  in  the  design  and  development  of  information  technology 
(IT)  platforms  for  government  and  university.  Dr.  Russo  has  a  Ph.D.  in  Agricultural  Meteorology  from 
Cornell  University,  Ithaca,  New  York,  M.Sc.  in  Meteorology  from  McGill  University,  and  a  B.S.  in 
Meteorology  from  St.  Louis  University.  Dr.  Russo  has  published  over  25  refereed  articles  in  professional 
journals,  and  has  been  a  columnist  for  Precision  Ag  magazine  since  2006. 

Mr.  Kevin  Marcus,  Marcus  Weather  Inc.,  Meteorological  Consultant 

Mr.  Marcus,  CEO  and  founder  of  Marcus  Weather,  is  a  meteorological  consultant  who  has  positively 
impacted  the  revenues  of  numerous  weather-sensitive  businesses  by  transforming  weather  data  and 
forecasts  into  easy-to-understand,  customized  reports  for  key  decision  makers.  He  has  developed  a  rule- 
based  weather  decision  tool  for  commodity  traders  that  has  revolutionized  the  way  that  industry  uses 
weather  data  in  their  decision  making  process.  Mr.  Marcus’  career  spans  more  than  30  years,  primarily 
serving  clients  in  the  energy  and  agricultural  commodity  industries.  His  meteorological  expertise 
combines  with  his  common  sense  approach  to  problem  solving  to  make  him  a  valuable  asset  to  the 
initiative  proposed  herein,  particularly  with  an  eye  toward  commercialization.  Mr.  Marcus  possesses  an 
M.S.  in  Agronomy  from  the  University  of  Maryland  and  a  B.S.  in  Meteorology  from  Penn  State  University. 


5. 5  Phase  II  Schedule  and  Milestones 

A  high-level  overview  of  our  planned  Phase  II  schedule  and  milestones  is  provided  in  Figure  18.  . 
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Figure  18.  Overview  of  Planned  Phase  II  Schedule  and  Milestones 


5. 6  COMMERCIALIZA  TION 

The  WIPCast  technology  proposed  herein  is  expected  to  garner  strong  interest  across  a  wide  array  of 
Government  agencies  and  weather-sensitive  businesses,  given  the  ability  of  resulting  analytical  tools  to 
quantify  and  better  anticipate  weather-related  risk. 

Specific  industries  to  be  targeted  include: 

•  Commodity  traders  would  benefit  greatly  from  the  ability  to  design  a  trading  program  around  the 
probability,  and  anticipated  severity,  of  weather  impacts  (similar  to  mission  planning);  similarly,  hedge 
funds  could  use  such  a  tool  to  exploit  anticipated  outlier  crop  production  conditions 

•  Energy  companies  and  utilities  could  employ  WIPCast  to  support  applications  that  manage  and  plan 
for  electric  demand  and  generation  requirements 

•  Food  manufacturers  would  benefit  from  the  ability  to  better  plan  for  price  risk  on  the  raw  materials 

•  For  the  travel  industry,  probabilistic  weather  impact  analysis  would  be  a  vital  asset  for  managing 
travel  risk  to  determine  the  best  route  or  time  window  of  travel 

Additional  anticipated  commercial  markets  and  applications  include,  but  are  by  no  means  limited  to, 
insurance,  transportation,  and  agriculture. 

Probabilistic  forecasts  are  an  idea  whose  time  has  come.  In  the  energy  industry,  for  example,  it  is 
estimated  that  hundreds  of  millions  of  dollars  could  be  saved  annually  through  use  of  improved 
probabilistic  forecasts  vs.  the  current  deterministic  forecasts  used  to  forecast  wind  generation,  energy 
demand  and  allocation  of  resources.  One  example  of  where  improved  forecasts  for  energy  demand  can 
be  gained  is  through  probabilistic  weather  forecast  inputs  into  the  load  models  used  to  predict  energy 
demand  across  the  electric  grid  system.  Deterministic  forecasts  are  routinely  used  that  include  no 
quantitative  measure  of  inherent  risk  in  the  forecast  of  temperature,  humidity,  sky  cover  and  winds.  A 
more  robust  approach  would  be  to  construct  an  ensemble  of  weather  inputs  that  will  quantify  risk  by  hour. 
Such  an  application  for  improved  short  term  forecasts  would  be  worth  millions  of  dollars  to  the  electric 
energy  industry. 

Similarly,  commodity  trading  and  purchasing  strategies  for  hedge  funds  and  food  manufacturers,  as  well 
as  electric  demand/generation  applications  for  energy  utilities,  could  generate  significant  WIPCast 
subscription  revenue  in  the  first  5  years  alone.  Also  for  the  travel  industry,  avoiding  poor  or  unsafe  travel 
conditions  translates  to  millions  of  dollars  of  revenue  each  year.  The  leisure,  commercial  trucking,  and 
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private  air  travel  industries  spend  $5-1 OMM  per  year  for  basic  weather  information.  A  decision  support 
system  that  combines  travel  risk  tolerance  with  uncertainty  in  weather  forecasts  could  itself  be  a  several 
million  dollar  per  year  market. 

Along  the  same  lines,  agriculture  businesses  spend  tens  of  millions  of  dollars  for  weather  data  and 
forecasts.  A  decision-based  product  already  provided  for  commodity  traders  and  food  manufacturers 
commands  two  to  three  times  the  industry  standard  for  private  weather  services.  Such  multiples  can  be 
applied  across  a  number  of  weather  sensitive  industries  once  the  WIPCast  algorithms  are  adapted  to  the 
context  of  the  economic  impacts  that  affect  these  other  businesses. 

Marcus  Weather  Inc.,  our  Phase  II  partner,  is  well  positioned  to  lead  the  market  penetration,  since  it  has 
already  established  a  beach  head  in  these  industries  through  30  years  of  providing  service,  developing 
new  weather  based  interpretative  products,  adapting  new  technology  for  rapid  assessment  of  short  term 
and  longer  term  weather  risk. 

With  regard  to  licensing  and  pricing,  we  at  Impact  Computing  are  strong  proponents  of  the  open  source 
software  licensing  model.  Consistent  with  this  philosophy,  WIPCast  is  to  be  licensed  and  distributed  as 
open  source  software.  It  is  important  to  note,  though,  that  our  belief  in  the  open  source  model  is  not, 
strictly  philosophical,  it  is  also  practical.  Open  source  has  clear  benefits  from  the  standpoint  of  market 
penetration.  Companies  that  offer  open  source  software  are  often  able  to  establish  an  industry  standard 
and  thereby  gain  competitive  advantage.  Many  companies  have  also  benefited  from  the  fact  that  the 
open  source  model  helps  build  developer  loyalty  as  developers  feel  empowered  and  have  a  sense  of 
ownership  of  the  end  product.  Moreover,  the  costs  of  marketing,  packaging  and  distributing  open  source 
technologies  are  typically  less  -  often  far  less  -  than  those  costs  for  more  traditionally  protected  and 
licensed  competitive  technologies. 

In  fact,  recent  years  have  witnessed  explosive  growth  in  the  success  of  open  source  technologies  in 
attaining  dominant  market  share  positions.  Examples  include  the  Apache  Web  Server,  the  Red  Hat 
distribution  of  the  Linux  operating  system,  the  MySQL  relational  database,  the  Mozilla  Firefox  web 
browser,  and  the  Sendmail  email  server.  Each  has  demonstrated  sufficient  stability,  capability  and 
maturity  to  capture  significant,  and  in  some  cases  dominant,  market  share,  even  against  alternative 
products  from  commercial  software  vendors  no  less  formidable  than  Microsoft  and  Oracle. 

Beyond  the  benefits  of  open  source  from  the  standpoint  of  market  penetration,  it  is  also  important  to 
recognize  that  open  source  products  can  and  do  produce  revenue  streams.  Red  Hat  and  MySQL  are 
prime  examples  of  companies  that  have  built  viable,  highly  successful,  commercial  businesses  based  on 
open  source  products  and  the  open  source  revenue  model,  whereby  revenue  streams  are  primarily  the 
result  of  product-related  subscriptions  and  custom  services.  WIPCast- related  revenue  is  therefore 
expected  to  be  generated  from  these  sources  as  well. 

WIPCast  development  beyond  the  initiative  proposed  herein  is  expected  to  be  accomplished  through  a 
combination  of  open  source  contributions,  internal  investment,  and  funded  enhancements  as  summarized 
in  Figure  19.  . 
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Figure  19.  Projected  WIPCast-related  Revenues  and  Investment 


29-Jun-2011,  vl.O 


Impact  Computing  Corporation 


Page  38 


SBIR  Topic  A10-065  /  Probabilistic  Forecasting  for  Aviation  Decision  Aid  Applications 

Phase  I  Final  Report 


6  Closing  Remarks 

The  combined  Impact  Computing  /  University  of  Washington  team  has  greatly  appreciated  the  opportunity 
to  support  the  Government  on  this  important  groundbreaking  project  in  the  domain  of  decision  support 
tools  that  exploit  the  power  of  probabilistic  forecasting. 

Our  recently  completed  Phase  I  initiative  has  been  both  challenging  and  rewarding  and  we  look  forward 
to  hopefully  proceeding  to  a  follow-on  Phase  II  effort. 
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